document 

24 



RESUME 



SE 004 940 



ED 024 573 2^ 

By- Smith, Herbert A- 

Institute on Evaluation for Science Programs- Final Report. 

National Science Teachers Association, Washington, D C- 

Spons Agency- Office of Education (DHEW), Washington, D-C. Bureau of Research. 

Bureau No-BR-8-8045 
Pub Date Apr 68 
Contract- OEC- 1-7-070006-3789 
Note- 1 18p. 

.Evalo..ion- -Models. Program Descriptions. Science Institutes. -Science 

IdeTtrers- aS: for the Education of Teachers of Science, National Scence Foundation. National 

Science Teachers Association, US Office of Education ^v,-|,,^tinn model- 

The institute was designed to provide intensive exposure to evaluatwn model 

and some important new evaluation techniques for persons P ^ a.<Hncation 
administrative, supervisory, consultant, or 

^eventv-severi oerson- psrticipsted in th6 Institute. A S6fY^3ntic Dif f 

ore- Ind Dort-^rtspon=e situa^ns was administered in an attempt to determine if 
^ignifSn.'^S.w/ shfu occurred coring .h. IrUiMe. P»r«g;; ^ S 

sheets were used and thoroughly reviewed by 

■Institute Evaluation- instrument was developed ®t"d administered a ^ the |^d o th 
Institute Participant evaluation of the institute was highly favorable^ Ihe analysis or 
responses "ffhe Semantic-Differential Scale demonstr^ated a ^ 
difl^rence between pre- and post-scores surpassing the .001 le 
a positive shift in attitude. (BC) 



Institute on Evaluation 
For Science Programs 



Project No. 3-3045 
Grant No. OEC-1-7-0 70006-3789 



Institute Director: Herbert A. Smith 



April 1968 



The research reported herein was performed pursuant to a 
grant with the Office of Education, U.S. Department of 
Health, Education, and Welfare. Contractors undertaking 
such projects under Government sponsorship are encouraged 
to express freely their professional judgment in the 
conduct of the project. Points of view or opinions stated 
do not, therefore, necessarily represent official Office of 
Education position or policy. 



National Science Teachers Association 



Washington, D.C. 



r 



Acknowledgments 



The Director is particularly indebted to Dr. Blaine Worthen, 
Dr. Robert Hammond and Dr. Albert Eiss . As senior instructors and 
team leaders, Dr. Worthen and Dr. Hammond played key roles in the 
success of the Institute. Their contributions during the planning 
phases were also invaluable. Dr. Eiss had large responsibilities 
for the initiation and preparation of the proposal and in the 
detailed planning for the Institute. He was also extremely 
helpful with the details involved in arranging appropriate physical 
facilities and in the day to day operation of the Institute. 



Herbert A. Smith 
Institute Director 



t { 





TABLE OF CONTENTS 



Page 

Introduction ^ 

Method ^ 

Results 3 

Discussion * ^ 

Conclusions ^ 

Appendix A: 

Announcement Letter 

Application Blank A-3 

Acceptance Letter 

Rejection Letter ^”5 

Program 

Appendix B: 

List of Participants Accepted B-1 

Occupational Classification and Geographical 

Distribution of Participants B-7 

Appendix C: 

Semantic-Differential Scale C-1 

Participant Daily Reaction Sheet C-2 

Institute Evaluation Instrument 

Appendix D: 

Perspective for Evaluation of Science Education D-1 

Evaluation — The Process of Providing Information to Aid 

in Decision Making . D-11 

Indicators of Change by Metfessel and Michael D-17 

Product Evaluation Worksheet #1 D-23 

Selected References in Evaluation D-25 

Simulation Problem Materials D-26 

Some Methods of Data Collection D-7& 

Appendix E: 

Appendix Table 1: "Responses by Teachers and 

Supervisors to Statements About the Institute E-1 

Appendix Table 2; Evaluation by Teachers and Supervisors 

of Specific Activities by Days E-2 

Analysis of Responses to the Semantic-Differential Scale . E-3 



o 



Introduction 



Science education has been characterized in recent years by 
many significant changes involving theory, curriculum, facilities, 
and instructional materials. Although innovations have been plentiful, 
there has often been little attempt to evaluate the effectiveness of 
such efforts and many changes can be viewed in retrospect to have 
been ill-advised. One of the major problems is the lack of 
individuals experienced in evaluating science programs. A large 
number of science supervisors have risen through the ranks of 
classroom teaching, with little or no specialized training for their 
new positions. College and university teachers who are retained 
as consultants for developing science programs are experienced in 
their subject matter fields, but often lack specialized training 
in education, particularly in evaluation. As a result, many of the 
evaluative procedures are carried out in a haphazard fashion and 
effective procedures are not identified and inefficient practices 
are not eliminated. 

The Institute was designed, therefore, to provide an intensive 
exposure to evaluation models and some important new evaluation 
techniques for persons employed in key administrative, supervisory, 
consultant or instructional roles in science education. More specifically 
it was anticipated that the Institute would contribute to the following 
objectives . 

1. Provide a small but effective group of individuals capable 
of making critical evaluations of science programs in 
local school districts. 

2. Provide a basis for further dissemination of evaluation 
techniques through follow-up workshops and inservice 
education at the local level. 

3. Provide materials and insights helpful in planning 
future national Institutes to upgrade evaluation of 
science education programs. 

4. Identify potential leaders in the use of evaluation 
procedures who could staff further institutes and inservice 
programs at the local level. 

5. Stimulate a more intensive and comprehensive approach 
to evaluation problems and further work in the field. 




Method 



In order to achieve the objectives of the Institute, it was 
decided to invite between 100 and 120 participants ^o were 
intensively involved with the problems of science education ei ^ 
arsuperviLrs of science education in the public schools or college 
lev^ triiners of science teachers. Individuals were selected from 
all levels of education, elementary through college. 

Following approval of the project, an intensive planni^ 
session was held with the Director, senior instructors, an WST 
representative, and one additional Institute staff member in 
attendance. Senior instructors and other 
members were drawn from the Evaluation Center at 0 i 
and from Project EPIC located at the University of Arizona. 

Specific plans were developed and a detailed program was formulated. 

Following the planning session, application blanks and an 
explanatory letter were sent to members of the Association for 
Education of Teachers of Science, the National Science Supervisors 
and to selected individuals known to be heavily involved in scie 
education programs. In addition, the NSTA made a separate mailing 
to supervisors and consultants in science education maintained on 
the NSTA Registry. In total, approximately 10,000 letters were 
sent. Because of the short time between the approval of the 
proposal and the dates for the Institute, the time 
the invitation letters and the Institute was short. (Between two 
and three weeks). In spite of the time limitation, 173 applications 
were received. (It should be borne in mind that attendance was 
entirely at the expense of the individual or of his employer.) Of 
these, 107 were invited to attend. Copies of the application 
blank, explanatory letter, acceptance letter, letter of regre 
those who could not be accommodated as well as the program mai 
to prospective participants are included as Appendix A. Seven y- 
seven participants were involved in the Institute, although a few 
were absent at either the beginning or the end of Institu e. 

Individuals who accepted but who did not attend the Institute a e 
indicated by the red circle around their identifying number in the 
roster of those invited in Appendix B. In addition to 
Institute participants, several observers were 

time. TheL represented the NSTA, the NSF and the U. S. Office of 
Education. One NSE participant was present for all sessions. A 
breakdown showing geographical representation and levels of pro- 
fessional responsibility is provided in Appendix B-7. 

In planning the Institute, it was evident that a program 
designed to sensitize participants to evaluation procedures and 
techniques should itself be a reasonable model with respect to the 
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evaluation process. Accordingly, considerable thought was given 
to the problem. It was decided to use a Semantic-Differential 
scale in pre— and post— response situations in an attempt to 
determine if significant attitude shifts occurred during the 
Institute. In addition, participant daily reaction sheets were 
used and these were collected and thoroughly reviewed by the 
Institute staff at the end of each day. Finally, a document, 
"Institute Evaluation" was developed and administered at the end 
of the Institute. Copies of the Semantic-Differential Scale, 
the participant daily reaction sheets and the final evaluation 
instrument are included as Appendix C. 

Some of the instructional materials which were used, much of 
it specifically developed for the Institute, are included as 
Appendix D. In addition, packets of materials, too bulky to be 
included as a part of this report, were prepared and distributed by 
the Project EPIC staff. One complete set of the materials is 
included with this report. 



Results 

The Institute was held in Crabtree Auditorium. The program^ 
ran very nearly as it had been planned . Evaluations of the Institute 
were analyzed using the computer facilities of Project EPIC. For 
purposes of analysis, participants were separated into "teachers 
and "supervisor" groups. The details of the analysis of the responses 
to the Semantic-Differential scale and to the Institute Evaluation 
instrument are shown in Appendix E. 



Discussion 

Fiom an examination of the participant evaluations (Appendix 
Table E) it can be seen that the responses to the Institute were 
highly favorable. It should be noted that the items in the first 
section of the Institute Evaluation Instrument (Appendix G) were 
not all polarized in the same direction. Thus, in items such as 
3, 28 and 41, "disagree" or "strongly disagree" responses constitute 
an endorsement for the Institute. 



The day to day evaluations of the specific activities are 
shown in Appendix Table F. There is strong evidence that the 
participants thought the program x^as reasonably balanced with the 
majority of responses falling in the ^*about right" category. 

There is evidence that more time should have been allowed on the 
second day program which was essentially concerned with the 
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simulation problem. There was a feeling among some of the 
participants that the simulation vjas either unrealistic or a 
"bad example" of problems in evaluation of science programs. 

(This viexvT x\ras not shared by the Director who thought it might 
quite possibly be just too realistic.) There was an indication 
that more work in refinement of the simulation problem and more 
time for participants might have paid large dividends. 

The daily evaluation forms were also helpful. From their 
analysis it became apparent that "evaluation" was being oriented 
too strictly to project assessment, particularly as these related 
to ESEA Title I and Title III activities. Comments on the evaluation 
sheets enabled the Institute staff to shift much more strongly 
in the direction of teacher, classroom and general program 
evaluation. 

The analysis of the responses to the Semantic-Differential 
scale demonstrated a highly significant difference between the 
pre and post scores surpassing the .001 level of significance 
in the direction of a positive shift in attitude. The analysis 
of the responses and pertinent comments are included as Appendix E 3. 

One participant tended to be highly critical and felt that 
the purpose of the Institute was misrepresented. He appeared to 
have expected specific ansx"7ers to such questions as whether CBA or 
CHEMS should be installed in his school. In any case, he expected 
a detailed evaluation of many of the specific science curriculum 
studies which have been funded. 



Conclusions 



The Institute appears from available evidence to have been a 
highly successful operation. It is apparent from the extremely 
short response time allowed betx-aeen the announcement letters and 
the deadline for applications that there is a felt need for assistance 
in evaluating science programs. There was evidence that the 
presentation of the CIPP evaluation model pifovided needed insights. 
Reactions of the participants seemed to indicate that, at present, 
context and input evaluation are likely to be particularly xjeak. 

More familiarity with process and product evaluation seemed evident. 

In any future Institute, it seems probable that the CIPP model and 
the simulation problem could and should be more closely related. 

The success of this Institute clearly indicates that future 
efforts of this kind should be planned. It is suggested that Institutes 
operated in connection with selected NSTA regional conferences would 
be especially appropriate. Institutes should not necessarily be 
limited to such meetings, however. 
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NATIONAL NSTA INSTITUTE IN EVALUATION OF SCIENCE EDUCATION 



Dear Colleagues in Science Education: 

An unusual opportunity is being provided for key personnel in science 
education to participate in a four-day institute on th«i evaluation of science 
teaching programs. The institute on evaluation of science instruction will 
be held just prior to the NSTA Annual Convention in Washington, D. C. You 
are invited to apply as an institute participant. The institute is sponsored 
by NSTA under a grant from the U. S. Office of Education. 

Presentations will be made by recognized authorities in evaluation. 
Participants will be provided with an opportunity to sharpen their insights 
into the evaluation process in a seminar setting. Objectives of the institute 
are: 

1. Suggest ways of evaluating plans for educational improvenent which 
include • 

a. Evaluation of a proposed strategy for improvement. 

b. Evaluation of procedures including provisions for feedback. 

c. Evaluation of outcomes of improvement programs. 

2. Selection of appropriate evaluation techniques designed to provide 
information necessary for making decisions about educational 
programs, classroom practices and curriculum changes. 

3. Development of ability to discriminate between well written and 
poorly written objectives given several sets of educational 
objectives of variable quality. 

4. Describe interaction analysis as a useful evaluative tool. 

5. Develop a well documented report of the project including a 
clear and concise evaluation of the institute. 

Time is very short and an almost immediate response is necessary from 
those who desire to participate. Below are the "facts" relating to the 
institute. If you are interested, please return your application form by 
return airmail. In any case, applications must be received no later than 
March 13. 

When ; The Institute begins at 9:00 a.m. , March 24 and ends March 27 
at 4:30 p.m. 

Where : All sessions of the Institute will be in Washington, D. C. 

Deadline : Applications must be received by March 13. Please use 

airmail . 

Fees : None. But participants must be supported by their employers 

or by themselves. 
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Staff : Dr. Blaine R. Worthen, Associate Director, Evaluation Center, 

The Ohio State University. 

Mr. Michael D. Hock, Research Associate, Evaluation Center, 

The Ohio State University. 

Dr. Robert L. Hammond, Director, Project EPIC, University of 
Arizona. 

Staff Associates, Project EPIC, University of Arizona 

Dr. Herbert A. Smith, Director of Teacher Education, Colorado 
State University. 

Eligibles ; Only 100 participants can be accommodated. Selection will 
be made from those applicants who are actively engaged 
as a supervisor or consultant for a local or state 
educational agency or who are employed by an institution 
of higher education and responsibile for supervision of 
science teachers, teaching of science methods course and/or 
direction of research in science education. 

Housing Accommodations : Arrangements have not been completed. However, 

NSTA will arrange for a block of rooms, probably 
at one of the Washington hotels, at the most 
favorable rate possible. 



Sincerely yours. 




203 Liberal Arts Building 
Colorado State University 
Fort Collins, Colorado 80521 
Telephone: 303-491-5305 
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APPLICATION FOR ACCEPTANCE INTO THE 
NATIONAL NSTA INSTITUTE IN EVALUATION OF 
SCIENCE INSTRUCTION 



Name : 



Address: Street 

City S tate ^Zip 

Employer ^ Present Job Title: 



Respond as appropriate in columns A, B, C, and D. 



A 



B 



C 



D 



Job Classification: 
I { Teacher 

i I Administrator 

I I Supervisor 

Q Other 



Responsibility as to: 

Grades taught: Q Full-time 

Level: (' [ Part-time 

Level : 

Level: 



Total educational 
experience: 
years 



Educational Background 

Bachelors Degree 

Institution: 

Date : 

Ma j or : 

Minor : 



Graduate Study 

Institution (s) 

Degrees, if any: 
Ma j or : 



Professional Affiliations 



What contribution x^ould you hope the NSTA Institute would make in helping you 
fulfill better your present responsibilities? 



If accepted, do you want housing accommodations made for you? Q yes i 1 no 

(Participants will be provided with information as to the housing of the 
Institute as soon as possible. This will probably be in a local hotel and 
the most favorable rates it is possible to obtain will be negotiated.) 

Best address to use in corresponding with you: 

Telephone number: (Include area code) 

Please note - Return application by airmail to: Dr. Herbert A. Smith 

203 Liberal Arts Building 
Colorado State University 
Fort Collins, Colorado 80521 
Telephone: 303-491-5305 
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Fort Collins, Colorado 
March 15, 1968 



Dear Participant; 

I am happy to inform you that you have been accepted as a 
participant in the National NSTA Institute in Evaluation of 
Science Education. A schedule of the program is included for 
your information. 

The Institute will be held in Crabtree Auditorium in the NEA 
Building located at 1201 Sixteenth Street, N. W. If you have 
requested housing reservations, they are being made for you at 
the Burlington Hotel. If you indicated on your application that 
you wanted reservations made for you, please fill out the slip 
stapled to this letter and mail to the hotel at once. The rates 
are $11 for single and $15 for double rooms. If you desire a 
double room, you should so state when you mail the attached form 
indicating the time of your arrival. Please note that it is your 
responsibility to notify the Burlington Hotel of your arrival 
time. 



As a reminder, participants should fully understand that 
attendance at the Institute is either at their expense or at the 
expense cJ their employer. No funds to defray any part of the 
cost of attendance are available through the Institute itself. 

We are looking forward to a stimulating and productive 
Institute and anticipate being able ' to greet you there* 



Sincerely yours, 




Herbert A. Smith 
Institute Director 
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Fort Collins, Colorado 
March 15, 1968 



Dear Colleague in Science Education: 

The response to the invitation to apply for the NSTA Evaluation 
Institute exceeded both expectations and the capacity to accommodate 
all who submitted applications. Consequently, it is with consider- 
able regret that your application cannot be honored. If the 
Institute is successful, it may be possible in the future to 
sponsor similar conferences in connection with NSTA regional 
meetings. Your name and application form will be retained as 
evidence of your interest and desire to participate in an Institute 
devoted to problems of evaluation and forwarded to the NSTA 
central office. 

Thank you for so promptly submitting your application. As 
you can probably easily conceive, the problem of selecting 100 
participants from all those who applied was not easy. Selection 
had to be done quickly and with the exercise of a comparatively 
high degree of subjective judgment. To the charge of subjectivity, 
the Director pleads guilty and begs your indulgence. 



Sincerely yours 




Herbert A. Smith 
Institute Director 



NATIONAL NSTA INSTITUTE IN EVALUATION OF SCIENCE INSTRUCTION 
Crabtree Auditorium, NEA Building 
1201 Sixteenth Street, N. W. 

Washington, D. C. 



March 24 



9:00 - 9:30 a.m. Opening Session 

Greetings and Announcements 
Introduction of Staff 



9:30 - 10:15 


a.m. 


"A Perspective for Evaluation of Science Education" 
Dr. Herbert A. Smith 


10:15 - 10:30 


a.m. 


Coffee Break 


10:30 - 12:00 


a.m. 


Evaluation as the Base for Educational Improvement 


12:00 - 1:30 


p.m. 


Lunch 


1:30 - 3:00 


p.m. 


CIPP Evaluation for Decision-Making 


3:00 - 3:15 


p.m. 


Coffee Break 


3:15 - 4:30 


p.m. 


Designing Evaluation Designs 



9:00 - 10:30 a.m. 
10:30 - 10:45 a.m. 



March 25 

Techniques for Gathering Evaluation Information 
Coffee Break 



Simulated Evaluation Design Prob lem in Science Education 

(remainder of the day) 



10:45 - 


12:00 


a.m. 


Focusing the Evaluation 


12:00 - 


1:30 


p.m. 


Lunch 


1:30 - 


3:00 


p.m. 


Collecting, Organizing, and Analyzing Information 


3:00 - 


3:15 


p.m. 


Coffee Break 


3:15 - 


4:30 


p.m. 


Administering, Reporting, and Interpreting 



Evaluation 



9:00 - 10:00 a.m. 



10:00 - 10:30 a.m. 



10:30 - 10:45 a.m. 



March 26 

Introduction to the EPIC Evaluation Concept, 

Model and Structure 
Dr. Robert L. Hammond 

Introduction to the Writing of Behavioral Objectives 
in Science 
Dr. Wayne Roberson 

Coffee Break 
A-6 
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10:45 - 12:00 a.m. Small Group: Objective Writing Session 

Richard Pov/ell, Drs . Terry Cornell, Wayne 
Roberson, Robert Kraner, and Robert Armstrong 



12:00 - 1:30 p.m. Lunch 



1:30 - 3:00 p.m. 



Small Group: Critique of Teacher -Developed 

Behavioral Objectives in Science, \7ith 
Discussion 



3:00 - 


3:15 


p.m. 


Coffee Break 


3:15 - 


4:00 


p.m. 


Small Group: 








Writing 


4:00 - 


4:30 


p.m. 


Small Group: 



Individual Participant Objective 
Critique of Participant Objectives 



March 27 



9:00 - 9:30 a.m. Introduction of Observational Systems for Teacher 

Self -Appraisal 

Drs. Terry Cornell and Wayne Roberson 



Characteristics of Observation Systems: 

a. Common Communication 

b. System for Tallying Behavior 

c. Procedures for Analyzing Behavior 

d. Skill Development 

e. Evaluation (comparison of pre-active 
phase of teaching with interactive 
phase .) 



9:30 - 10:30 a.m. Observational System for Coding Teacher Objectives, 

Methods, and Expressions Utilizing Video Tapes 
and Computer Analysis--Dr . E. Wayne Roberson 

10:30 - 10:45 a.m. Coffee Break 



10:45 - 12:00 a.m. 

12:00 - 1:30 p.m. 

1:30 - 3:00 p.m. 

3:00 - 3:15 p.m. 

3:15 - 3:45 p.m. 

3:45 - 4:15 p.m. 

4:15 - 4:30 p.m. 



Flander's Interaction Analysis, Description of 
Coding Process, Emphasizing Use of Matrices and 
Feedback--Dr . Terry Cornell 

Lunch 

Small Group Session: Resource People Available to 

Discuss Morning Presentations and Their 
Utilization in the Evaluation Process 

Coffee Break 

Response to Institute by Selected Participants 

Evaluation of Institute (Individual written 
responses) 

Closing Remarks and Adjournment 
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PARTICIPANTS 

NATIONAL NSTA INSTITUTE IN EVALUATION OF 
SCIENCE INSTRUCTION 
March 24-27, 1968 




Anderson, Harold M. 
Professor of Education 
University of Colorado 
377 Ord 

Boulder, Colorado 80302 



10. Brawley, Joanna 

Elementary Science Consultant 
Ferguson-Florissant School District 
815 January Avenue 
Ferguson, Missouri 63135 



2. Bachus, Ralph E. 

Science Supervisor 

Boulder Valley School District 
2550 Linden Avenue 
Boulder, Colorado 80302 

3. Balassone, James M. 

Coordinator of Science 
Great Neck Public Schools 
866 Middle Neck Road 
Great Neck, New York 11024 

4. Bedel le, Adrienne G. 

Science Consultant 

City School System of New Rochelle 

Stephenson Boulevard 

New Rochelle, New York 10801 

5. Binger, Robert D. 

Science Consultant 

Florida State Department of Educ. 

1601 Raa Avenue 
Tallahassee, Florida 2303 

6. Bleecker, Anthony L. Jr. 

Dean of Science Instruction 
Pennsbury School District 
25 Crown Terrace 
Morrisville, Pennsylvania 19067 

I 

7y Bloom, Harold W. 

Supervisor of Science 

Board of Education of Anne Arundel Co. 

1602 Ebbotts Place 
Crofcon, Maryland 21113 



Buell, Robert R. 

Professor of Science Education 
University of Toledo 
Toledo, Ohio 43606 

12. Burke, Richard J. 

Science Supervisor 

Greenhills -Forest Park Board of Ed. 
72 Drummond Street 
Cincinnati, Ohio 45218 

13. Carter, Joseph 

Science Consultant /Science Teacher 
Anaheim Union High School District 
1047 Buckeyewood 
Orange, California 92667 

Chimento, Russell L. 

Program Specialist, Science 
Sacramento City Unified School Dist. 
6654 13th Street 
Sacramento, California 95831 

15. Corbusier, Edith 
Director of Education 

Cleveland Heights -University Heights 
Board of Education 
2155 Miramar Boulevard 
Cleveland Heights, Ohio 44121 

16. Cunningham, John 
Director of Curriculum 
Mansfield Board of Education 
R199 Pulver List Road 
Mansfield, Ohio 44905 



8* Bolles, William H. 

Science Education Adviser 
Department of Public Instruction 
Box 911 

Harrisburg, Pennsylvania 17126 




D'Ambrosio, Nicholas 
Assistant Professor of Science 
Paterson State College 
26 Dixon Place 
Wayne, New Jersey 07470 



9. Bonney, Catherine Y. 

Science Supervisor 
Newark Special School District 
83 East Main Street 
Newark, Delaware 19711 



28. DeSeyn, Donna E. 

Science Consultant and Director 
Earth-Space Science Education Ctr. 
9 Fifth Avenue 
Fairport, New York 14450 
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9. Eastman, Thomas W. 

Director of Science 
Needham Public Schools 
58 Hawthorne Avenue 
Needham, Massachusetts 02192 

Eide, Edwin 

Science Chairman and Teacher 
Fenton High School, Bensenville, 111. 
930 Elm Street 
St. Charles, Illinois 60174 

1. Fails, Donald J. 

Science Supervisor 
Gateway Schools 
386 Kenney Avenue 
Pitcairn, Pennsylvania 15140 

2. Farmer, Walter A. 

Chairman, Science Education Dept. 
State University of New York, Albany 
R. D. Box 152 

East Berne, New York 12059 

Flannery, Edward J. 

Coordinator of Science 
Council Bluffs Community Schools 
23 Arnold Avenue 
Council Bluffs, Iowa 51501 

h. Ford, Eleanor M; 

Associate Professor of Physics 
Fairmont State College 
453 Callen Avenue 
Morgantown, West Virginia 26505 

Forgie, Amelia E. 

Supervising Teacher 
Green Bay Board of Education 
100 N. Jefferson 
Green Bay, Wisconsin 54301 



28. Gentry, Adrian N. 

Coordinator of Instruction 
Director, ESEA Title III 
Science Project 

County Superintendent of Schools 
Box 868 

Riverside, California 92502 

29. Gibson, Carolyn A. 

Director, Univ. of Pittsburgh 
High School Science Research Prog. 
Lecturer in Education 

187 Moreland Road 
Pittsburgh, Pennsylvania 15237 

30. Given, Thomas D. 

Coordinator of Science 
Birmingham Public Schools 
632 Ardmoor 

Birmingham, Michigan 48010 

31. Haney, Richard E. 

Associate Professor of Science Educ. 
University of Wisconsin, Milwaukee 
4321 N. 42 Street 
Milwaukee, Wisconsin 53216 

32. Hanson, Claude A. 

Science Supervisor 
Boise Public Schools 
3417 Kelly Way 
Boise, Idaho 83704 

33. Harbison, Fay 

Director, ESEA Space Science 
Learning Program, Federal Project 
Administrator 

Newport-Mesa Unified School District 

543 Tustin Avenue 

Newport Beach, California 92660 



Fors, George 

Science & Mathematics Consultant 
Department of Public Instruction 
Bismarck, North Dakota 58501 

Fox , Gene T . 

Science & Mathematics Supervisor 
Prince William County School Board 
5313 Garner Street 
Springfield, Virginia 22151 



Harris, Kandel 

Science Consultant 

Northeast Georgia Instructional 

Services Unit 

320 Irvin Street 

Cornelia, Georgia 30531 

(35y Hazel ton, Ralph 

Science Coordinator 
Dover Special School District 
134 Reese Street 
Dover, Delaware 19901 
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Heath, Elbert C. 

Science Coordinator 
Consolidated School District #2 
R. R. #1 

Oak Grove, Missouri 64075 



45. Kilburn, Robert E. 

Science Coordinator 
Newton Public Schools 
46 Marked Tree Road 
Needham, Massachusetts 02192 



Hudgins, William Kent 
Science Consultant 
PACE Education Center 
Wood County Schools 
2109 42nd Street 

Parkersburg, West Virginia 26101 

Hunt , Agnes 
Science Supervisor 
Birmingham Board of Education 
2015 7th Avenue North 
Birmingham, Alabama 35203 

Jachimowicz, T. J. 

Curriculum Development Specialist 
Department of Public Instruction 
166 North Clay Street 
Manheim, Pennsylvania 17545 

Jenkins, Jack L. 

Secondary Science Chairman 
Utica Community Schools 
53076 Ruann Drive 
Utica, Michigan 48087 

Jones, Frances D. 

Science Consultant 
Alabama Department of Education 
2919 N. Colonial Drive 
Montgomery, Alabama 36111 

Jones , Thomas D , 

Science Coordinator 

Maine-Endwell Central School District 
633 Lacey Drive 
Endwell, New York 13760 



46. 




48. 



49. 



50. 





Keegan, Mary E. 

Science Consultant 
Winnetka Public Schools 
1111 Spruce Street 

Winnetka, Illinois 60093 53. 



Kellogg, Maurice G. 

Associate Professor of Science Educ. 
Western Illinois University 
287 Jana Road 
Macomb, Illinois 64155 



King, Marjorie M. 

Science Consultant 
Jefferson Parish School Board 
519 Huey P. Long Avenue 
Gretna, Louisiana 70053 

Kirkbride, Robert D. 

District Science Representative 
Napa Valley Unified School District 
1106 Larkin Way 
Napa, California 94558 

Kleinman, Gladys S. 

Associate Professor 
Hunter College 
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High School Principal 



20 

6 

33 

3 

10 

22 
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GEOGRAPHICAL DISTRIBUTION OF PARTICIPANTS 
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weak 
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strong 


Interesting 
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boring 


unfair 
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clean 
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dirty 


sharp 
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dull 


important 
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1 


unimportant 


sour 1 1 


1 1 sweet 


cold 1 1 


1 1 hot 


good 1 1 


1 1 1 bad 


worthless 1 1 


1 1 1 valuable 


meaningful 1 


1 1 meaningless 


long 1 


[ 1 1 short 


distasteful 


i 1 1 tasty 
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PARTICIPANT DAILY REACTION SHEET 



Session Date 



A. Your questions (about content, facilities, etc.) 



B. Your comments (on content, presentation, instruction, 
facilities, etc.) 



C. Your suggestions (regarding content, instruction, arrangements, 
etc.) 
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NATIONAL NSTA INSTITUTE IN EVALUATION 
OF SCIENCE INSTRUCTION 
WASHINGTON, D. C. - MARCH 24-27, 1968 



I-N-S-T- 


I-T-U-T-E 


e-v-a-l-u-a-t-i- 


-0-N 


appropriate 


boxes: 


- 




Teacher 


r? 


Elementary 


£7 


Supervisor 


r? 


Junior High 


Z7 




Senior High 


o 






College 


n 



Below are a numbei? of statements concerning objectives and certain 
teacher self-approach techniques. You are to indicate how much you 
agree or d isagree with each of the statements by encircling the letter 
representing one of the following expressions. 

Strongly Disagree (SD) ; Disagree (D) ; Neither Agree nor Disagree (N) ; 

Agree (A) ; Strongly Agree (SA) 



1 . 



I now have a better idea of what evaluation is "all 
about" than I had before this institute. 



D N A SA 



2. Evaluation plays a critical role in educational 

improvement. SD D N A S 

3. The evaluation concepts and techniques presented 

in this institute have little relevance to evalu- 
ation problems I am likely to face in the future. SD D N A SA 



4. Looking at types of decisions (planning, program- 
ing, implementing, and consequential) is a useful 
way to begin to focus on the type of evaluation 
information which is needed. 

5. I feel that I could identify types of decisions 
which need to be made in most science programs 
with which I might work. 

6. The CIPP (context, input, process, and product) 
evaluation model is a useful way to view evalua- 
tion of science programs. 

7. Sorting the "lists of decisions" (on day 1 of the 
institute) into types was a useful technique to 
illustrate differences among decision types. 



SD D N A SA 



SD D N A SA 



SD D N A SA 



SD D N A SA 



8. It is important to do context and 
tion before deciding on a program 
action. 



input evalua- 
or plan of 

SD D N A SA 
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3. In general, I feel that I would know how to conduct 
context evaluation in planning a science program. 

0. In general, I feel that I would know how to conduct 
input evaluation in selecting from among alternative 
programs, etc. 

1. In general, I feel that I would know how to conduct 
process evaluation in monitoring program activities. 



SD 



SD 

SD 



2. In general, I feel that I would know how to conduct 

product evaluation in relating outcomes to objectives. SD 



.3. The structure for developing evaluation designs is 
useful in attempting to design an evaluation for a 
science program. 

L4. I feel I could use the structure for developing 
evaluation designs to design an evaluation which 
met minimal evaluative criteria. 

L5. Many of the techniques identified on the second day 
of the institute (e.g., interviews, unobtrusive 
measures, achievement tests, etc.) are relevant for 
evaluation in science programs. 

L6. I believe that I personally could use most of the 
techniques if they seemed relevant. 



SD 



SD 



SD 

SD 



L7. I believe I understand when the varying techniques 
might be appropriate. 

18. The simulated evaluation design problem was useful 
in giving me a feel for how one might go about 
designing an evaluation. 

19. The feedback in the simulation was helpful to me 
in understanding the design process. 



20. The theory behind Roberson's Self -Appraisal System 
is good, but is unrealistic in real life. 

21. Developing objectives makes me feel more confident. 

22. I feel a sense of insecurity when attempting to 
learn a code. 



SD 

SD 

SD 



23. Learning a code makes me feel as though I'm lost 
in a jungle of numbers and can't find my way out. 

24. Objective writing is something which I enjoy doing. 

25. I become confused and unable to think clearly when 
learning to code . 



D N 

D N 
D N 
D N 



D 



N 



D N 



D N 
D N 
D N 



D N 

D N 

D N 
D N 

D N 

D N 
D N 

D N 



A 



A 

A 

A 



A 



A 



A 

A 

A 



A 

A 

A 

A 

A 

A 

A 

A 



SA 



SA 

SA 

SA 



SA 



SA 



SA 

SA 

SA 



SA 

SA 

SA 

SA 

SA 

SA 

SA 

SA 




C-4 



-3- 



26. 



27. 

28. 

29. 

30. 

31. 

32. 



I feel that Flander’s Interaction Analysis is not ade- 
quately designed to provide useful evaluatron 
inforinatlon. 

When I hear the «ord objective, I have a feeling of 

dislike , 

Coding a teacher’s behavior does not serve a useful 
purpose. 

The development of program or educational objec- 
tives is a necessary procedure. 

Objective writing is too complicated to learn in a 
one -day workshop. 

Flander’s Interaction Analysis is good in theory, 
but is unrealistic in real life. 

Beginning teachers are too inexperienced to write 
objectives. 



33. 

34. 



35. 

36. 



37. 

38. 



39. 



40. 



I feel coding is very useful. 

I feel a positive reaction toward the ideas presented 
at the conference. 

I become frustrated when I think about writing 
objectives. 

I approaeh writing l^^f^/norieingik 

hesitation resulting from a tear oi noL ucj. g, 

in writing objectives. 

I feel at ease when learning to code. 

Roberson’s Self -Appraisal System is adequately de- 
sl^e^to provide useful evaluation information. 

neveloDing good program objectives and instructional 
objective! !ill facilitate the improvement of teach- 
ing procedures. 

The objectives of this institute were not the same as 
my objectives. 



SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


SA 


SD 


D 


N 


A 


S/ 



41. 

42. 

43. 

44. 

45. 



I could have learned as much by reading a book. 

The instructors really knew their subject. 

The daily schedules were too fixed. 

There was too much lecture and too little interaction. 



SD 

SD 

SD 

SD 

SD 



D 

D 

D 

D 

D 



N 

N. 

N 

N 

N 



A 

A 

A 

A 

A 



SA 

SA 

SA 

SA 

SA 



46. 



I think I would be under a terrible strain when coding 
teachers . ' 

Objective writing is very important to me. 
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The major topics which were presented in this institute are lifted below. 
Would you please respond to each topic by checking whether you think 
time spent on it was too much, too little, or about right. 



During the institute , the time 
spent on this topic was: 



(check one) 



FIRST DAY 


Too Much/About Right /Too Little 


1 . A perspective for evaluation of 
science education (Smith) 


□ 


□ 


a 


2 • Evaluation as the base for 
educational improvement 
(Worthen-Hock) 


□ 


□ 


a 


3, CIPP evaluation for decision- 
making (Worthen-Hock) 


□ 


□ 


a 


4, Designing evaluation designs 

(Worthen-Hock) 


□ 


□ 


□ 


SECOND DAY 


Too Much /About Right/Too Little 


1 . Techniques for gathering 
evaluation information 
(Worthen) 


□ 


a 


a 


2* Simulation: Designing context 
evaluation (Worthen-Hock) 


O 


□ 


□ 


3. Simulation: Designing input 
evaluation (Worthen-Hock) 


a 


□ 


□ 


4. Simulation: Designing process 
evaluation (Worthen-Hock) 


□ 


□ 


□ 


5, Simulation: Designing product 

evaluation (Worthen-Hock) 


[□ 


□ 


a 



i ERIC 
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THIRD DAY Too Much /About Right /Too Little 



1 . EPIC evaluation model 
(Hammond) 


C 


3 


□ 


□ 


2 . Writing and critique ing 
behavioral objectives 
(Arizona team) 


L 


3 


□ 


□ 


FOURTH DAY 


Too Much 


About Right 


Too Little 


1 . General observational systems 
for teacher self-appraisal 
(Cornell-Roberson) 


• 

r 


3 


□ 


□ 


2 . Coding teacher behavior: 
Video tape and computer 
analysis (Roberson) 


r 


□ 


□ 


a 


3 . Coding teacher behavior: 

Flander's interaction analysis 
(Cornell) 


r 




□ 


□ 


4 . Evaluation of institute (what 
you're now doing) (Smith) 


f — 
\ 


□ 


□ 


□ 


Total time spent in the institute was: 


i: 




□ 


□ 
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A PERSPECTIVE FOR EVALUATION OF SCIENCE EDUCATION 

Herbert A. Smith 
Director of Teacher Education 
Colorado State University 

It is certainly a pleasure to welcome a group of science 
educators to a conference on the evaluation of science instruction. 

The fact that the response to the invitation to the Institute was 
as substantial as it was on the very short notice provided is an 
index of the current interest and the felt needs which exist for 
a more adequate and effective approach to the problems of evaluation. 
The objectives of this meeting were spelled out in some detail in 
the invitation which went to you along with the announcement of 
the Institute. However, I think it can be simply stated that the 
real purpose 6f the Institute is to determine how science instruc- 
tion can be improved and how it can be made to have the highest 
degree of relevancy for all students who are enrolled in the 
elementary and secondary schools. Evaluation is a key element in 
any such improvement. The concept guiding this Institute is that 
evaluation covers a total system: students, teachers, supervisors 

and administrators, administrative organization, facilities— in short 
all those things that impinge directly or Indirectly upon the 
climate for learning. 

During the Institute we will have some highly qualified 
experts in the area of evaluation who will work with you in a 
variety of situations. We believe that the concepts and materials 
which they present will prove to be highly beneficial. But in 
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fairness to them, I think I should state that they do not 
represent themselves as science educators, rather they are 
evaluators. Consequently, there are some things that they cannot 
do. There are specific questions related to science education 
that they are not qualified, and cannot be expected to be 
qualified, to answer. Many of these questions are philosophical 
in nature and require certain value judgments which fall more 
specifically within the province of science educators. To 
illustrate, they cannot tell us what we ought to be doing in 
science education. During their presentation they will raise 
questions, I presume, which will be pertinent to the decision- 
making process which is involved in curriculum and program 
development, but ultimately these decisions are the responsibility 
of science educators and administrators. To put it another way 
I would not anticipate that they will answer for us "Where should 
we be going, and why?" As I understand their role, they expect 
to be able to tell us "how to go there" more effectively. They 
will help us identify our possible failures along the way and 
perhaps help us realize when we can reasonably be said to have 
arrived at our destination. I hope they will have something to 
say about the relativistic character of educational objectives. 

Essentially, my role in this Institute is to serve an 
administrative and coordinating function. However, I should 
like to step aside from these assignments for a few minutes to 
speak as a science educator and to exercise a director's 
prerogative by commenting upon the current scene in science 
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education. There is a substantial basis for believing that a 
conference devoted to the subject of the evaluation of science 
education is critically imperative at the present time. In the 
next few minutes I hope to be able to raise some questions and 
issues X 7 hich seem rather vital to science education. 

Education has been characterized in recent years by an 
immense amount of activity and science education has exhibited 
its full share. But as I have observed the passing scene, our 
educational establishment often seems like a ship without a 
rudder or compass but which is, nevertheless, cruising x-7ith all 
engines set at full steam ahead. It reminds me of the legendary 
Texan who dashed into the Dallas airport and demanded a ticket 
from the first airline ticket clerk that he encountered. She 
said, "Well sir, will you tell me where you want a ticket to?" 
and he said, "Listen, lady, it doesn't make any difference, I've 
got business everywhere." 

It is high time that our fundamental purposes for science 
instruction receive an intensive and critical examination. I seem 
to detect no unified consensus as to the direction in which we 
should head. It is obvious that we need vision as well as skills, 
wisdom as well as knowledge, and humanity as well as competence, 
and that our educational system has a major responsibility for 
seeing that such aspirations are attained. Too often we see 
a strange hiatus between the existence of knowledge and the logical 
implications which knowledge has for action. We have intellectual 
accomplishments not coupled in any way with productive behavior 
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and personal or social responsibility. Such intellectual 
sterility may reflect a lack of any real commitment and be a 
reflection to a large degree of our failure to ansx^er in a 
convincing manner, "’O'Jhere are we going?” 

Vast sums of money have been spent in developing curriculum 
materials without any very consistent guide as to what ends will 
be served by the materials produced. Great masses of materials 
have been produced in recent years and most of them have reflected 
an "elitist philosophy.” The emphasis has definitely been on 
subject matter content. The educational axiom that content should 
be selected in terms of the needs of people has been too cavalierly 
abandoned. Students often complain bitterly today about the lack 
of relevance of much of their education for their lives and for 
the real world. I have said in print elsewhere that I am convinced 
that materials produced by the large scale curriculum projects 
which have been so generously funded are largely irrelevant for 
more than half of the students in our schools. And I think that 
that statement is probably far too conservative. It may be 
closer to being irrelevant for 80-85% of our students. I hope 
that our evaluators will have something to tell us about the 
kind of standards and expectations which might be related to 
children and adolescents. T^at is a "criterion of reasonableness" 
for educational attainment which should be applied in an educational 
setting? There are substantial evidences today that too many 
children are being pushed too far, too fast, and too hard. 
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It is clearly no secret to many of you that science courses, 
and most especially physical science courses, do not enjoy high 
popularity with either high school or college students. Part of 
this unpopularity may well be related to the question of grading 
and evaluation. It has always been a mystery to me as to why 
students usually expect to receive lower grades in science and 
in mathematics than in other subjects. Often, their expectations 
are completely fulfilled. One strongly suspects that a certain 
amount of academic snobbery is the culprit and many science 
teachers, both in secondary schools and colleges, seem to think 
that their function is to flunk students rather than to teach 
them. Avoiding science classes except for the minimum required 
for graduation may be evidence of good judgment on the part of 
many gifted students who really cannot risk a ”C" or "B" grade 
if they wish to contend for scholarships or other awards or win 
entrance into a name university. This statement in itself, of 
course, may be indicative of some of the serious ailments in 
our educational system. No large campus or school is likely to 
be free of the pompous pedant who loudly proclaims that 'Nobody 
gets an A in my class!" The tragedy of this situation is that 
the clown expects to be rewarded because of the high standards 
he maintains. One might maintain that either his standards are 
unreasonable, or he is a poor teacher, or both. 

The signs of alienation of youth need no lengthy elaboration 
here. They are all about us. The statistics on any number of 
indices are available which show the heavy impact of failures 
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in schools and in homes; the failures of teachers and parents 
to meet the basic needs of youth. To list but a few indices 
would include the present adolescent suicide rate, the incidence 
of mental illnesses, various psychosomatic afflictions, juvenile 
crime rates, dropouts, "hippyism," teenage drug addiction and 
teenage prostitution. Nor are the alienated restricted to 
minorities, the poor, or the socially and culturally disadvantaged. 
Suburbia and the "upper -uppers" are also well represented. Youth 
is in rebellion and the lack of relevancy of their education to 
their perceived needs and interests is plainly evident. 

One always hesitates to deride the current emphasis on 
subject matter for when he does he is so often interpreted as 
saying that subject matter is unimportant. Obviously, what is 
really needed is more and not less subject matter but it affords 
us little benefit if, in trying to teach the subject matter, we 
"successfully" alienate the great mass of adolescents. The 
real issue is to teach functional subject matter in such a way 
that students recognize and accept its validity and utility. 

One of the reasons why we have been relatively unsuccessful 
in reaching many students is to be found in the fact that most 
of the new programs have been almost exclusively concerned with 
the cognitive aspect of education and especially with abstract 
concepts. The affective domain of attitudes, appreciations, 
hopes, and aspirations have too often been ignored. In fact, 
often its existence seems not even to have been recognized. 
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The young do not bring the same motivations to the classroom 
that scientists bring to their laboratories and yet it would 
appear that the scientists in their enthusiasm have often 
projected themselves and their own motivations and interests into 
many of the materials which they have produced. They have 
forgotten that the views they hold and the satisfactions they 
now seek are quite unlike those that they possessed when they 
vjere 12, 14 and 16 year old youngsters. The past, particularly 
of successful men, is often viewed through the rosiest of tinted 
lenses and selective memory and nostalgia recreate a scene bearing 
little resemblance to reality which once existed. 

The affective domain cannot be ignored for two very 
substantial reasons. In the first place, it is in the affective 
domain that we find the driving forces which innervate and 
direct the energies of youth. Second, it is in the affective 
domain that we find those qualities which make men, men, rather 
than animals. For the great mass of human beings, science will 
always be important in terms of its meaning for human life and 
society and for only a very few will it be a source of intellectual 
delight and adventure. Consequently, science cannot be taught 
in a vacuum apart from its social, political and economic 
derivatives. Considering the vast forces which the scientists 
have unleashed it is certainly in the interests of all that 
those destined to be scientists, and who are now enrolled in our 
elementary and secondary school classes , should be imbued with 
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a compassion for humanity, with a concern for larger problems, 
and a perspective of the place of science in the world including 
some awareness of its potential for good or evil. 

I am sure that in this Institute you are going to hear 
something about "behavioral objectives." It is certainly 
pertinent that you should, for the aim of all education is surely 
to both change behavior and provide the potential for future 
changes in behavior. Yet, I have to smile when I hear the 
new gospel preached and listen to the new cliches which extol 
the ultimate virtues of "behavioral objectives." We find in 
Ecclesiastes 1:10: 

Is there anything whereof it may be said, see this is new? 

it hath been already of old time, which vjas before us. 

Nearly a century ago Charles Sanders Pierce was learnedly 
and profoundly debating on "operational theory of meaning" 
with William James and others, holding that the meaning of a 
concept was essentially incorporated in a set of operations 
(behaviors). If one accepts Pierce's thinking it is surely not 
a long step to the contention that educational objectives can 
have no meaning except in terms of behaviors. In essence, 
to speak of behavioral objectives is a tautology. I recall in 
my own classes of 20 and more years ago of asking students how 
the class had changed their behavior and disturbing them by the 
assertion that my course and all the other courses they had 
taken were worthless to the extent that they had not changed 
behavior or created a potential for changed behavior in future 
situational contexts. 
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But I do not particularly intend to detract from the current 
emphasis on the "behavior” aspect of educational objectives. 

Perhaps such emphasis is long overdue. However, I do not think 
we can identify all those behaviors which are likely to be 
pertinent. We again run the risk of miring in a swamp which 
Ernest Bayles calls "specific objectivism.” It is really a 
question of the generality of objectives which is involved. 

Just how detailed does one become in listing the desired behavior 
outcomes? 

With the present emphasis on behavior there is danger that 
we will get bogged down with behavioral minutia and lose sight 
of those broader behavioral objectives which are the real goals 
of education. It is by no means certain that competence in small 
behavior units will necessarily add up to those broader behavior 
patterns which determine the basic life patterns xdiich lead to 
a good and productive life. Hopefully, a part of our Institute 
program will bring a degree of realism and practicality to our 
attempts to assess ouir progress in the modification of behavior 
and that it will bridge a dangerous gap by helping us steer 
between the shoals of behavioral minutia and the reefs of over- 
extended and meaningless generality. 

I suspect that there will be aspects of the science education 
program in xjhich evaluation is sorely needed which will not be 
touched upon in any adequate way in this Institute. For years, 

I have felt that there is an inadequate definition of the role 
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of the science supervisor. The wide variety of titles and 
responsibilities assigned is a reflection of this fact. The 
relationships between system supervisors and building principals, 
the hierarchy of administrative controls, the processes of 
planning and budgeting, the informal infra-structure of the 
system and many other factors have a direct and sometimes deadly 
influence on the science instructional program. Important as 
they are, it seems probable that we will not enter into these 
areas in any depth. I mention these here only to insure that 
they are not overlooked or thought to be unimportant. 

Problems of evaluation are among the most complex in all 
of education. Consequently, we should not be so presumptuous 
as to expect this Institute to answer all the questions which 
might legitimately be raised. Perhaps our functions will have 
been well served if we are able to increase your sensitivity to 
problems of evaluation, provide you with some enlarged insights 
and the opportunity to interact among yourselves, demonstrate 
some special techniques and stimulate you to think seriously 
about evaluation as it relates specifically to problems within 
the context of your owi system responsibilities. We expect it 
to be a working . conference and we hope it will be a productive one. 
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EVALUATION = The Process of providing information 

to aid in decision-making. 



LACKS IN EDUCATIONAL EVALUATION 



1. Lack of adequate definition of and validated theoretical 
bases for evaluation. 

2. Lack of knowledge about educational decision processes 
and information requirements. 

3. Lack of appropriate evaluation designs. 

•+. Lack of appropriate evaluation instruments, tools, 



and techniques. 

5. Lack of integrated mechanisms for organizing, processing, 
and reporting evaluation information. 



THESIS: A major purpose of evaluation is to provide information 



RATIONALE : 

1. Quality education demands continuing efforts to improve 



education; 

2. Educational improvement requires an appropriate balance of 
enlightened persistence and change; 

3. Obtaining and maintaining this delicate balance requires 
sound decision-making; 



M-. Sound decision-making depends upon an appropriate supply 
of evaluative information. 



6. Lack of trained evaluation personnel. 



EVALUATION FOR DECISION-MAKING 



for making decisions. 
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COMPARISON OF EXPERIMENTAL AND EVALUATION STRATEGIES 



experimentation EVALUATION 



POSE 


To test research hypotheses 


To facilitate the continual 
improvement of a program 


JECTS 


Units to be measured are 
randomly assigned to treat- 
ment and control conditions 


Subjects are assigned to a 
program based upon their needs 
and the purpose of the program 
rather than the requirements 
of the data collection and analy- 
• sis designs 


TROL 


Treatment and control condi- 
tions are held constant 
throughout the experiment 


Evaluation aims to stimulate 
improvement in on-going programs 


RUMENT 

NISTRA- 


Instruments are administered 
after a specified period of 
time--usually a year or some- 
times pre and post to the 
experiment 


Instruments are administered 
to conform with information 
requirements of decision- 
makers throughout the pro- 
gramme existence 


BACK 


Avoided during the experiment 
so as to avoid contamination 


An essential means for stimu- 
lating change 


ERIA 


Information should be: valid, 

and reliable 


Information should be: valid, 

reliable, timely, credible, and 
pervasive 



DECISION-MAKING IN EDUCATIONAL IMPROVEMENT 



PROBLEM: To evaluate for decision-making, the relevant decisions 

must be known. 

POSTULATE: Decisions in educational Improvement activities may be 

classified as:. 

Planning (focusing needed improvement activities) 

Programing (specifying procedure, personnel, facilities, budget, 
and time requirements for implementing planned 
activities) 

Implementing (directing programed activities) 

Recycling (terminating, continuing, evolving, or drastically 
modifying activities) 
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Daniel L. Stufflebeain 



DE VELOPI NG EVALUAT ION D ESIGNS 



The logical structure of evaluation design Is the same for all types 
of evaluation, whether context, Input, process or product evaluation, T e 
parts, briefly, are as follov/s: 



B. 



Focus Inq the Evaluation _ ^ ^ 

T, ^Identify the major level (s) of decls lon-maklng to be served, 

e.g., local, state, or national, ^ ... 

2, For each level of dccls lon-maklng, project the decision 

situations to be served and describe each one In terms of Its 
locus, focus, criticality, timing, and composition of alterna- 

3. Define criteria for each decision situation by specifying vari- 
ables for measurement and standards for use in the judgment of 
alternatives, 

4. Define policies v/lthln which the evaluation must operate. 



Col lection of Information 

1, Specify the source of the Information to be collected, 

2, Specify the Instruments and methods for collecting the needed 

information, 

3. Specify the sampling procedure to be employed. 

4. Specify the conditions and schedule for Information collection. 



C. Organization of information ^ ^ . n i 

1, Provide a format for the Information which Is to be collected, 

2, Designate a means for coding, organizing, storing, and re- 
trieving Information. 



0. Analysis of Information 

1. Select the analytical procedures to bo employed. 

2. Designate a means for performing the analysis. 

E. Reporting of Information 

1, Define the audiences for the evaluation reports. 

2. Specify means for providing Information to the audiences. 

3, Specify the format for evaluation reports and/or reporting 
sessions, 

4 . Schedule the reporting of Information. 

F, Administration of Evaluation 

1, Summarize the evaluation schedule, 

2. Define staff and resource requirements and plans for meeting 
these requirements, 

3. Specify means for meeting policy requirements for conduct of 

the evaluation. . ^ -j* 

4 , Evaluate the potential of the evaluation design for providing 
Information which Is valid, reliable, credible, timely, and 
pervasive. 

5. Specify and schedule means for periodic updating of the 
evaluation design, 

6, Provide a budget for the total evaluation program. 



KINDS OF EVALUATION FOR DECISION-MAKING 



PROPOSITION: Each class of decisions in educational Improvement 

activities requires a relevant kind of evaluation. 

Context evaluation is for planning decisions and provides 

information about ’’what needs to be done? 

Input evaluation is for programing decisions and provides 

information about ’’what can be done?” 

Process evaluation is for implementing decisions and provide 

information about ’’what is being done?” 

Product evaluation is for recycling decisions and provides 

information about ’’what has been done?” 
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Y7illlam B* Michael/ UCSB 



I. Indicators of Status or Change. in Cognitive and Affective. Behaviors of. 

Students in terms of Standardized Measures and Scales ^ 

• : Standardized achievement and ability tests, the scores on which allow in- 

ferences to be made regarding tlie ^extent to which cognitive objectives 
concerned with knowledge, comprehension, understandings, skills, and 
applications have been attained. 

m . J 1 ^ . * • 

Standardized self inventories. designed to yield measures of adjustment, 

.. . appreciations, attitudes, interests, and temperament from which inferences 
can be formulated concerning the possession of psychological traits (such 
as defensiveness, rigidity, aggressiveness, cooperativeness, hostility, 
and anxiety). 

Standardized rating scales and check lists for judging the quality of 
products in visual arts, crafts, shop activities, penmanship, creative 
writing, exhibits for competitive events, cooking, typing, letter writing , 
fashion design, and other activities. 

Standardized tests of psychomotor skills and physical fitness, 

II. Indicators of Status or Change in Cognitive and Affective Behaviors of 
Students by Informal or Semiformal Teachermade Instruments or devices 

Incomplete sentence technique: categorization of types of responses, 
enumeration of their frequencies, or ratings of their psychological 
appropriateness relative to specific criteria. ' - . ~ 

Interviews: frequencies and measurable levels of responses to formal and 

informal questions raised in a face-to-faqe interrogation.. 

Peer nominations: frequencies of selection or of assignment to leader-, 
ship roles. for which the spclogram technique may be particularly suitable. 

Questionnaires: frequencies of responses to items in an objective format 
and numbers of responses to categorized dimensions developed from the 
content analysis of responses to open-ended questions. 

Self-concept perceptions: measures of current status and indices of con- 
gruence between real self and ideal self--often determined from use of the 
semantic differential or Q-sort techniques, 

» . ^ • • ' . * . 
Self-evaluation measures: student's own reports on his perceived or desired 
level of achievement, on his perceptions of his personal and social adjust- 
• . ment, and on his future academic and vocationail plans. 
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Teacher-devised projective devices such as casting characters in the class 
play, role playing, and picture interpretation based on an Informal scoring 
model that usually embodies th^ determination offreguehcles of the occur- 
rence of specific behaviors , Br ratings of their Intensity or quality i 

'■ Teacher-made 'achievemerit tests (objective and essSy), the scores on which 
allow Inferences regarding the extent to which specific instructional 
objectives have been attained*' ^ ^ .* 

Teacher-made rating scales and check lists for observation of classroom 
'behaviors: performance levels of speech, music, and art, manifestation 
' ' ''of creative endeavors, personal and social adjustment > physical well being# 

’ Teacher-modified forms (preferably with consultant aid) of the semantic 
differential scale# ‘ 



III. Indicators of Status or Change in Student Behaviors Other than Those 
Measured by Tests, Inventories, and Observation Scales in Relation to 

■ the Task of Evaluating Objectives of School Programs ‘ — 

Absences: full-day, half-day, part-day, and other selective indices 
pertaining to frequency and duration of lack of attendance# 

Anecdotal records: critical incidents noted including frequencies of * 
behaviors judged to be highly undesirable or highly deserving of commendation# 



Appointments: frequencies with which they are kept or broken# 



Articles and stories: number’s and type published in school newspapers, 
magazines, journals or proceedings of student organizations. 



Assignments: ■ riumfaers and types cbmpleted with 'sbme' sort 6f quality rating 



or mark attached • ^ 



Attendance: frequency and duration when attendance is required or con 
sldered optional (as in club meetings, special events, or off-campus 
activities)# 



Autobiographical data: behaviors reported that could be classified and 
subsequently assigned judgmental values concerning their appropriateness 
• relative to specific objectives concerned with human development. 

Awards, citations, honors, and related indicators of distinctive or 
creative performance: frequency of occurrence of judgments of merit in 
terms of scaled values. 



Books: numbers checked out of library, numbers renewed, numbers reported 
read when reading is required or when voluntary# 
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Case histories: critical incidents and other passages reflecting quanti- 
fiable categories of behavior. 

Changes in program or in teacher as requested by student: frequency of 
occurrence# 

Choices expressed or carried out: vocational/ avocational/ and educational 
(especially in relation to their judged appropriateness to known physical/ 
intellectual/ emotional/ social/ aesthetic/ interest/ and other factors) • 

Citations: commendatory in both formal and informal media of communication 
such as in the newspaper/ television/ school assembly/ classroom/ bulletin 
' board/ or elsewhere (see Awards) . 

"Contacts'': frequency or duration of direct or indirect communications 
between persons observed and one or more significant others with specific 
reference to increase or decrease in frequency or to duration relative to 
selected time intervals . 

Disciplinary actions taken: frequency and type . 

Dropouts: numbers of students leaving school before completion of program 
of studies. 

Elected positions: numbers and types held in class / student body/ or out- 
of-school social groups , 

Extracurricular activities: frequency or duration of participation in observable 
behaviors amenable to classification such as taking part in athletic events/ 
charity drives/ cultural activities/ and numerous service— related avocational 
endeavors. . ' . 

Grade placement: the success or lack of success in being promoted or 
retained; number of times accelerated or skipped. 

Grade point average; including numbers of recommended units of course work 
in academic as well as in non-college preparatory programs , 

Grouping: frequency and/or duration of moves from one instructional group 
to another within a given class grade. 

Homework assignments: punctuality of completion/ quantifiable judgments 
of quality such as class marks. 

Leisure activities; numbers and types of; times spent in; awaras and prizes 
received in participation. . 

Library card: possessed or not possessed; renewed or not renewed. 
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Load; numbers of units or courses carried by students/. 

.Peer group participation: frequency and duration of activity in what are 
judged to be socially acceptable and socially undesirable behaviors. 

Performance;, awards, citations received; extra credit assignments and asso- 
: ciated points earned; numbers of books or other learning materials taken out 
of the library; products exhibited at competitive events * 

Recommendations; numbers of , and Judged levels of favorableness, . 

Recidivism by students: incidents (presence or absence or frequency of 
occurrence) of a given student* s returning to a probationary status, to a 
detention facility, or to observable behavior patterns judged to be socially 
undesirable (intoxicated state, dope addiction, hostile acts including 
arrests, sexual deviation) , . , . . “ .. 

Referrals: by teacher to counselor, psychologist, or administrator for 
disciplinary action, for special aid in overcoming learning difficulties, for 
behavior disorders, for health defects or for part-time employment activities. 

Referrals; by student himself (presence, absehce, or frequency) , 

. Service points:, numb.er^s earned, .. . . , 

Skills: demonstration of new or increased competencies such as those 
found in physical education, crafts, homemaking, and the arts that are 
. not measured in a. highly .valid fashion by available tests and scales 



Social mobility: numbers of times student has moved from one neighborhood 
to another and/or frequency which parents have changed job's*, 

•. I*** 9 ••• 

Tape recordings; critical incidents^ and other analyzab^ events 

amenable to classification and enu'mefatibh, ' 

hardiness: fr%uency > 

...Transiency; incidents, of, . . 

Transfers: numbers of students entering school from another school 
^ .(horizontal move) , . . , ,, . .. 

Withdrawal; numbers of students withdrawing from school or from a special 
. program (see Dropouts) , •. ^ ... 

IV, Indicators of Status or Change in Cognitive and Affective Behaviors of 
Teachers and Other School Personnel in Relation to the Evaluation of 
School Programs, 



o 
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Articles: frequency aird types of articles and written documents prepared 

by teachers for publication or distribution. 

Attendance- frequency of, at professional meetings or at in-service training 

fessional person's desire to improve his competence. 

Elective offices: numbers and types of appointments held in professional 
and'social organizations. 

Grade point average: earned in postgraduate courses. 

Load carried by teacher: teacher-pupil or counselor-pupil ratio. 

Mail: frequency of positive and negative statements in written correspondence 
. about teachers , counselors , administrators , and other person 

Memberships including elective positions held in professional and community 
organizations: frequency and duration of association. 

Model congruence index: determination of how well the ®^ions pf pr^^^^^ 
fessional personnel in a program approximate certain operationally stat 
judgmental criteria concerning the qualities of a meritorious p g 

Moonlighting: frequency of outside Jobs and time spent in these activities 
by teachers or other school personnel. 

Nominations by peers , students , administrators or parents for outstanding 
service and/or professional competencies: frequency of. 

Rating scales and check lists (e.g., graphic ®oales or the^ 

differential) of operationally- stated dimensions of teachers behaviors in 
the classroom or of administrators' behaviors in the ^ . ^ 1 , 3 ^ 

skills attitudes, adjustment, interests, and work efficiency, the percep 
tlons of various members of the total school community (parents, teachers, 
administrators, counselors, students, and classified employees) of the 
behaviors of other members may also be obtained end compaied. 

Records and reporting procedures practiced by administrators, counselors 
and teachers: judgments of adequacy by outside consultants. 

Termination: frequency of voluntary or involuntary resignation or dismissals 
of school personnel. 



o 
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'Transfers: frequency of requests of teachers to move from one schpol to 



another . 

V. Indicators of Community Behaviors in Relation to the Evaluation of 
School Programs ... 



Alumni participation; nunibers of visitations, extent of involvement in 
PTA activities , amount of support of a tangible (financial) or a sery ce 
^ ‘'nathre to a continuing school program or activity,/ 



Attendance at special school events, at meetings of the board of education, 
or at other group activities by parents; frequency of. 



Conferences of parent-teacher, parent-counselor , parent-administrator 
sought by parents; frequency of request. 

Conferences of the same type sought and initiated by school personnel; 
frequency of requests and record of appointments kept by parents. 

Interview responses amenable to classification and quantification. 



Letters (mail); frequency of requests for information, materials, and 
servicing. . • v ' ' - 

Letters: frequency of praiseworthy or critical comments about schpol pro- 
grams and services and about the personnel, participating in thein, . 



Participant analysis of alumni; determlnatlpn of locale of graduates, 
oecupatlon , affiliation with paHicuiar In^^^ dr opt side agencies . 

^ • . . 

Parental response to letters and report cards.upon written or oral.reqviest 
by school personnel: ; frequency by 'parehts . , . 

• . *'*'.*.»*• * 

•Telephon'd baiis'from parents ^ aliiihni, arid frbrh.petso^ 

fnddia (e.g., newspaper reporters)V frequency, duration, .aiid quantifiable 
judgments about' statements mdhitbred from telephone conversations , 

Transportation requests; frequency' of , 
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SCHEDULE PRE- AND POST- ANALYSIS OF GAIN SCORES- 

TEST AND ADMINISTER COMPUTER PROGRAM 

PROVIDE OBSERVER, PRINT FREQUENCY COUNTS OB USE 

CODING FORMS, SET REPORT BY TEACHERS— OBSERVERS 

DEADLINE 
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In July of 1967, you accepted a position as a secondary-school science 
supervisor* During your first week on the job, you discovered that 
the district Curriculum Study Committee on Goals for the Sciences had 
formulated a new set of educational goals* The superintendent of 
schools asked each science supervisor to analyze existing curricula 
within his area of responsibility and to design, implement, and 
evaluate pilot curricula* Based on the results, recommendations were 
to be made concerning possible system-wide adoption of these curricula* 

You decided to focus initially on the biology curriculum* During the 
course of the year, you found that the following decisions had to be 
made in order to fulfill your charges 

1* Does the superintendent really want curricula to conform to the 
prescribed "Goals for the Sciences"? (i*e*, what are the goals 
he really -Wahts to attain?) 

2* Does the selected set of goals really address itself to problems 
and needs in your system? 

3* What are the goals of the present biology curriculum? 

4* How much divergence is there between the "new goals" and the 
;^^esent goals of the biology curriculum? 

5* What should the objectives of the biology curriculum be? 

6* Can we attain all of these objectives? 

7* How can we attain the objectives we have selected? 

8* What portions of the present biology curriculum are relevant? 

9* What alternative curriculum approaches might be used? 

10* What are the benefits and costs of each alternative? 

11* What criteria should we use in selecting from among the alternatives? 

12* Which alternative biology curriculum is best suited to our needs? 

13* Does operation of the selected curriculum create unforeseen 
procedural problems? 

14* Are the teachers able to teach the biology curriculum which is 
implemented? 

15* Does the selected biology curriculum yield the desired result — 
i*e*, does it attain the stated objectives? 

The above decisions are listed in the sequence in which they had to be 
made* Group these decisions into the four categories of decisions 
(planning, programing, implementing, and consequential) by drawing 
heavy lines at appropriate points to separate the decisions in the list 
into four groups* Label each group according to type of decision* 




DEC-1 
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You have recently been hired to aid the scxence 

unsuccessfully attempted to develop and evaluate a P^l^t currxcul 
in biology. Together you successfully culminated that task. 

In view of the difficulty in accomplishing the prior task, 
the supervisor have attempted to anticipate the decxsxons w x 
be made as you engage in similar analysis of the physxcs P»^o 9 ramr 
Through communication with administrators and xnstructxonal , 

you have identified and compiled a list of decxsxons for whxch 
evaluative information is needed. This list follows : 



1. What is the effect of participating teacher attitude on the new 
program? 

2. What means are available which will likely reduce the cost of 
material development for the curriculum? 

3. What similar types of new programs have other school systems 
attempted to implement? 

4. What level of academic achievement would we have expected in physics? 

5. Has student perception of the laboratory instructor affected new 
program operation? 

6* How might the sessions in the curriculum best be presented 
sequentially? 

7. Who among the students might most profitably be focused upon in 
this new program? 

8. To what degree does the proposed program appear likely to be 
administratively feasible for physics teachers to operate? 

9. What alternative interpretations exist for explaining the low 
level of understanding of physics concepts before we begin any 
new programs? 

lO. How will the proposed curriculum likely benefit students not 
enrolled in it? 



11. What type of attitude toward the "discovery” method should students 
manifest? 

12 . Are students enrolled in the new program given the allotted 
individual counseling time by faculty members? 

13 o What are the benefits of the program— what objectives have been 
attained? 

14. What is the current status of the problem which motivated our 
creating this new physics program? 
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15. What aspects of the in-service program for teachers of physics 
ought to be continued during the next year? 



The decisions listed above are not listed in sequence — they 

are not listed in the order in which they must occur for efficient 
program development. As before, your task is to categorize . 

decision by type (pl^anning, programing, implementing, and consequential) 

Label each decision. 



Because of your background in evaluation, you recognize that there is 
a direct relationship between the type of decision to be made and the 
type of evaluation (context, input, process, and product) needed to 
aid the decision-maker. 



DEC -2 
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SIMULATED EVALUATION DESIGN PROBLEM 



The enclosed is the first of several sets of simulated materials which will 
be given to you as part of a simulated evaluation design problem^ in science 
education. This simulation is based in part on an actual ESEA Title III 
scl©nc© proj©ct 

■ * However, much of the simulation material is totally, fk^titio u^. 

Many of the events portrayed herein never actually occurred* Names are 
fictitious, and many roles and behaviors are complete fabrications* In 
short, while an attempt has been made to lend an air of reality to the 
simulation by basing it on a real program, gross liberties have been taken 
in order to protect anonymity and to enhance the utility of the simulation* 

The following agencies have kindly allowed the use of their "letterheads 
to simulate correspondence related to the simulated project: 




should be stressed that all correspondence printed on these letterheads 
for this institute is simulated and bears no direct relation to any actual 
correspondence from any of these agencies. 
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As of now, you are E. Val Layfor, new Evaluation Specialist 
- ^ ' / ' / " ' ' ' * . You have just 

arrived at your office for the first time, having only recently left your 
former position as a science supervisor in a neighboring district. 

The materials attached here were left on your desk by Dr. Judd 
Mentle, former incumbent in this position. Unfortunately, you were 
unable to have a personal conference with him before he left for his new 
job. We suggest that you study carefully the materials which he left for 
you. YOU may need or want other specific kinds of information. If so, 
ask your secretary. Miss Rhea Sorse, and she can tell you whether or not 
such information is available . ^he simulation instructors will play the 
role of Miss Sorse — address any information requests to them.J 
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June 25, 1967 



MEMO: 




TO: 


Dr. E. Val Laytor 


FROM: 


Dr. Judd Mentle 


RE: 


Attached materials 



I regret that I have to leave before you arrive to replace me . I had hoped 
to be able to introduce you personally to some of the problems you will 
face as you assume the position of evaluation specialist. You 11 arrive at 
a critical time. The district has been under fire from several sources, 
particularly from a vocal citizens' group that feels we are "behind the 

times . " 

This spring, the board of education asked Superintendent McBride to 
organize a curriculum study committee to see if the district might analyze 
the report on "Regional Educational Goals for the 1980 s prepared by 
AEL (Appalachia Educational Laboratory, Inc.). A committee (they've given 
themselves the fancy title of "Goals Task Force") was formed and is 
working on recommendations now. If my guess is correct, they will have 
us changing everything in the system. If that happens, you 11 probably 
wish you'd stayed in Huntington! 

Supt. McBride was very strong with me on one point. Maybe you won't 
relate to him in the same way, but he made it clear to me that he was the 
decision-maker and my role was only to provide him with the information 
he asked for. 

If you get in trouble, rely on Miss Sorse . She is the best secretary I ever 
had. And she really knows the school system. I've had her collect some 
background information I thought you might find useful . The first piece is 
a section out of an unsuccessful Title I proposal we submitted. It will 
give you a quick idea of what the system is like . The rest of this stuff 
is self-explanatory. 

Good luck to you. 
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TiTLE I PROPOSAL 



The Community Setting 

There are striking and extreme contrasts which are apparent in the 
educational and cultural needs of the citizens At on® 

end of the spectrum are the professional and technical personnel who make up 
thirteen per cent of the work force. These citizens desire for themselves 
and their families the highest quality in educational and cultural opportunities. 
They participate In and support the efforts of many community organizations. 
(However, a Museum, Scouts, Campfire Girls, and kindred groups 

are the only community-supported organizations that offer enrichment programs 
to the community as a whole.) On the other end of this spectrum are 

■ communities so remote that their peoples can spend whole 
lifetimes without ever leaving These citizens have few. If any, 

desires for education or culture. The local one or two room school often 
represented the only culture and education they knew. Since It has been 
necessary to close all but a few of these schools through a program of con- 
solidation, many communities have no cultural or educational Influence 

remaining in their community lives. 
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average of $483. Such a low expenditure cannot allow the Inclusion of needed 
enrichment programs and activities for either the culturally deprived or highly 
gifted students, *both of which are found within the school system. 

An estimated 6% of the four and five year old children In the county 
participate in public school kindergarten or pre-school programs conducted 

with ESEA Title ! and Operation Headstart funds. 

The elementary classroom teacher to pupils ratio is 29:1. The secondary 

ratio is 22:1. 

The elementary professional staff to pupils ratio is 25:1. The secondary 
ratio is 19i2. 

The total enrollment In the school system over the past five years is as 
fol lows : 



1963- 64 - 60,010 

1964- 65 - 59,709 

1965- 66 - 59,664 

1966- 67 - 58,768 

projected 1967-68 - 57,559 



These figures show a decreasing enrollment pattern which reflects to some 
extent the 7% state population loss noted in the I960 census. No other major 

changes in enrollment have occurred. 

School facilities are not seriously overcrowded, and the current 
$31,900,000.00 building program, to be completed In 1971, will further improve 
school facilities. None of the 156 schools have been declared unsafe. 

is presently supporting the public schools at the maximum 
tax rate allowed by the State Constitution. The County is fully bonded 
(including a $22,900,000 bond issue voted in 1965 for capital improvements and 
a supplementary bond issue of $9,000,000 which was approved December I6, 196?) 
and the citizens have voted a 100% excess levy. The monies derived from the 
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excess levy are primarily used for instructional materials and to pay higher 
teachers* salaries. But salaries are still well below the national average. 

The only avenue remaining to Increase local support of the public schools 
Is through increased property evaluation. The present evaluation of real and 
personal property is at approximately 50% of true and actual value, which is 
the amount required by State law. Recently a group of influential citizens 
representing industry, business and the general public met to discuss the need 
for more adequate school financing. They voted to take the necessary steps 
to cause property evaluation in the county to be raised over the next two to 
five years. 

The county qualifies for assistance under the federally affected areas 
legislation. At present, no monies have been received, but full application for 
them is being made during the coming fiscal year. 
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from The Shiloh Times-Pispatch, March 1, 1967: 



NOTtO PHYSICIST TO AUDRESS 
. LOCAL SCHOOL OFFICIALS TODAY 

f 

( " MARCH 1 — The noted physicist Dr, Charles 

Isaiah, Director of the Educational Studies Center of the 
University of Chicago, will speak today on "Technology, 
Space , and the Man of the Next Century" before a group 
of local educators. Dr. Isaiah is an outspoken advocate of 
redirecting and remaking the academic curriculum in to ay s 
secondary schools. Mr. Moore of the 

Board of Education invited Dr, Isaiah after hearing his com- 
ments on the report "Regional Educational Goals for the 
1980's." Mr, Moore stated he "hoped Dr. Isaiah would give 
our local schools the direction which can come only from an 
expert involved in both an academic area ana the field of 
education in general . " 



BAC 1.3 



D-35 



MINtJTaSSi 



The February 16, 1967 meeting of the Board of Education was held in the base- 
ment of the Administration building from 7 to 10 o'clock; all members were 
present. Mr. Gidney began the meeting by a call for tabling the agenda in 
order to permit open discussion of the report "Regional Educa -^nal G a 
the 1980's, " prepared by the Appalachia Educational Laborato' : . It was 
Mr. Gidney' s belief that community reaction to the report's contents and 
spirit had been so deep and widespread that Board discussion ,;as warranted. 

The motion passed. 

Chairman Moore stated he was certain that all present f ® 

for action in the future which were implicit in the report. -le noted that the local 
reaction to the report would, in his opinion, be most valuable if undertaken in 
the spirit which surrounded the passage of the federal Elementary and Secondary 
Education Act of 1965 (especially as expressed in Title Hi). 

Mrs. Merriman observed that she had received many "creative suggestions" 
from close friends among the membership of the PTA; she believed that many 
of these ideas (especially concerning cultural enrichment in the Art Museum 
or the use of honors programs) might be fruitfully investigated by the Board or 

th© school administration. 

Mr. Wallbrown differed with Mrs. Merriman, noting that many of the ® 

constituents" were less articulate than members of the PTA but had problems 
perhaps more severe and wants just as strongly perceived as did students with 
whom Mrs. Merriman was acquainted. Several teachers in the audience stated 
that they believed remedying shortcomings in the existing program (notably 
by "updating" the academic program at the secondary level) would be a more 
profitable undertaking than "trying to solve the problems of students who are 

not born yet . " 

Tames Wagner suggested that the Board solicit recommendations from the 
administration and the community concerning steps which might be taken to 
respond to the challenges implicit in the Goals report. Chairman Moore pu 
the suggestion in the form of a motion and the motion passed unanimous V* 

He directed the secretary to circulate the minutes of this meeting to interested 

parties as a solicitation. 

Respectfully submitted, 

Tanet Baker, Secretary 
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May 20, 1967 



To the Members of the Board of Education 



Amidst all the concern for planning for the 1980's, the school 
must not overlook the wise use of available resources during present days 
The importance of integrating existing programs is equally as great as 
the importance of coordinating the development of new programs. For 
these reasons, we believe it is crucial to report to the Board on the 
operation of the program in the Planetarium 



There presently is little or no relation between use of the Planetarium 
by school personnel and students, and activities in the classroom prior 
to or after visits to the Planetarium. Yet the Planetarium appears to e 
viewed by school personnel as a valuable resource. There has been constan 
growth in the use of the facility, and a corresponding growth in the level 
of experience available to the visiting student. 



Prior to November, 1962, the facilities of The Children's Museum were 

housed in three rooms ° 

rooms held a Spitz Model A-1 planetarium projector and canvas dome where 
public lectures were given once a week and school lectures by appointment. 
This Planetarium was operated by volunteers under the supervision ot a 
semi-professional museum director. 



During 1962, a move to larger quarters enabled the Museum's 

Board of Directors to plan for a larger, better-equipped Planetarium. 

A capital fund drive was conducted and the sum of $35,000 was raised. 

A Spitz Model A-3-P projector, 20 foot perforated aluminum dome, reclined 
seating for sixty persons, and basic auxiliary equipment were purchased 
and installed. A small exhibit area adjoining the Planetarium was fur- 
nished with astronomical displays. The program was placed under the 
direction of a Museum Board member with amateur experience in the field, 
assisted by volunteer lecturers. 
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To the Members of the Board of Education 



Page 2 
May 20, 196? 



In 1964 the Museum Board asked the Schools to provide a 

teacher to serve as director of the Planetarium. Superintendent McBride 
expressed Interest In coordinating a planetarium Instruction with the 
regular school curriculum. However, funds were not available at that 
time nor In the foreseeable future to provide a teacher-director. Subse- 
quently, the Museum employed a part-time, non-professional curator to 
supervise the Planetarium’s operation and develop a specific program of 
lectures with the assistance of the science supervisors. 

One presentation for the primary grades has been completed to date, as 
well as an outline of the subjects to be covered by presentations for 
the higher grade levels. 



Adjacent counties frequently send students to 

Planetarium. During 1965-66, there were approximately 1,500 student 
visitors from adjacent counties attending In class groups. in addition, 
class groups from more remote counties amounted to about 1,450 student 
visitors. Little has been done, however, to integrate the use of the 
Planetarium into the curriculum of the schools. A trip to the Planetarium 
is viewed as a field trip — an enriching, but not a teaching, experience. 
How can the schools plan profitably for the 1980's w^;en the attitude of 
school personnel Is to so wasteful ly misuse existing resources? We are 
willing to work with the school system, and especially Its science super- 
visors, in creating an integrated program for Planetarium use; indeed, 
the availability of ESEA Title ill funds makes such a proposal practically 
an expense-free risk. Would it not be more rational to work toward the 
educational goals of the 1960‘s, before worrying about the 1980’ s? 



Sincerely yours. 






Board of Directors 



Task 1.1 
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PROGRAM ON COMMUNITY ACTION 

. _ . T '■ 

FOR TODAY’S YOUTH 
AND TOMORROW'S CITIZENRY 

June 9, 1967 



MEMORANDUM 

TO: The Board of Education 

RE: Solicitation of Reaction to "Regional Educational Goals For 
The 1980's" . 

In spite of the fact that _ is a major industrial 

complex and that average local family incomes are among the 
nation's highest, cultural deprivation, provincialism, and 
poverty are widespread* There are extreme contrasts evident 
among students, families, and communities in the 
. ^ - valley region. While industrial wages (a m.ajor part of 

the region’s economic base) in the region rank fourth highest 
in the country, and per capita income is ranked high by com- 
parison to the national average, twenty-one, per cent of the 
region's families subsist on incomes of less than^^ _i_0M.« 

There is no room among this latter group for the purchase of 
cultural or educational opportunities; all that is available to 
them are opportunities presented by the public school system, 
other government agencies, or private welfare agencies or 
charities. Most certainly you must not forget the youth of 
these families as you look at the "average" student of 



Gertrude Henry, Chairman. 



Sandra Raley 
Erwin McAuly 
Wilfred Sanders 
Charles Bloom 
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Task 1.2 



INDUSTRIALIST’S ASSOCIATION 



Office of the Director 



April 1S6? 



Dr. Philip McBride^ Supt 



Dear Phil; 

Just a short note to give you a rundown on the meeting we had over 
lunch today. A1 Herbst had just seen the Regional Goals report^ and 
was really excited by possibilities he savi in it for action by the 
school district. What would you say to running a "Science and the 
Vforld of V/ork" program through the grades using v/ork-oriented science 
materials, and having an adult education program in conjunction with 
that program which we would run in the assembly centers of our plants 
You could relate simple and complex principles in biology, chemistry, 
and physics to illustrate the operating technology of industry in 
the area, and use our equipment, facilities, and other resources to 
complement your science program. It seems to me, as it^did to 
that a curriculum like this would really meet the changing needs of 
kids in our region for an understanding and appreciation of both 
science and technology. 

A program like this would work into the program changes which you are 
undertaking (I understand), and also could advance the level of 
skill which our employees (both present and future) bring to the job. 
Of course, we'd be glad to work over coffee V'/ith you and your staff 
on this, and give some thought to providing books and perhaps some 
additional money for the program. I told Al, "This is the kind of 
thinking that can make this region greati" 



Sincerely yours. 




Task 1.3 
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^U^fC "tv ^\j^\ ^lux* 

^'liu- cH'^ur Li0fS>> 
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June \1 , 1967 



MEMO: 

i 

TO: Members of the Central Administration and the Science 

Department Faculty 

FROM: Ray Boroff and Peggy Bresnen, Science Study Committee 
Co-Chairmen 

RE: Suggestions for an Integrated Science Program to Meet the 

Needs of District Students 



Considerable interest has been aroused by the report "Regional Educational 
Goals for the 1980'Sc" The report has identified changes in the population 
and economic characteristics of the region likely to occur by 1990, and 
discussed implications of such changes for education and the impact of 
education on these characteristics. What stands out most clearly to 
us is a theme, recurring in the report, of the probable misuse of human 
and material resources in the region. Persons young and old will be 
underemployed, undereducated, and unprepared to meet the changing stresses 
of the coming years. The countryside will be more and more polluted. 
Workplaces will often be marginal economic undertakings, little understood’ 
and likely to be perpetuated in the region, greatly affecting an unsuspecting 
population . 

In a time of change , in a region where the quality of human and material 
resources is frighteningly low, it seems that the most flexible and 
effective response to new needs that our system can make is to produce 
healthy and adaptable people. Young and old need proper nutrition, proper 
dental hygiene, reasonable personal health habits, and positive personal 
mental health, to be ready to adapt. In addition, a knowledge of the 
capability of people to adapt and their biological and physical environment 
is a necessary requisite for making successful and lasting adaptations. 
Finally, a knowledge of processes which make the human system and the 
ecological system operate as they do is a necessary requisite for operating 
successful adaptation, both personal and social. 



. D-43 



TO: Members of the Central Administration and 
the Science Department Faculty 



Page 2 
June 17, 1967 



It is our belief that an integrated science program can meet the most 
pressing need of our region in the coming years --the need to know and 
understand the process and product of adaptation. Biological constructs, 
physics concepts, principles of chemistry and the earth sciences can 
all be used to instruct the student and give him understanding, and can 
all be related to the student’s person, his life style, and the environment 
in which he lives . It is our belief that the science faculty ought to move 
forward to propose and plan a science curriculum which will help our 
students adapt, and which will have meaning for them. A unigue oppor~ 
tunity for meeting the real needs of our students lies within reach. What 
is your reaction? Will you work with us on such a program? 



Task 1 .4 
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Contents of 

Student Cumulative Record 
Folder 



1. Student Name, Address, Telephone, Parents' Names 

2. Birthdate, Sex 

3. Parents' Occupation, Education (years completed) 

4. Brothers and Sisters Presently Enrolled in System 

5. Grade Average in Subjects by Year, and Teacher 

6. IQ Scores (California Test of Mental Maturity)— grades 6, 10 

7 . Standardized Achievement Test Scores 

Stanford Achievement Battery grades 3, 6, 9, 11 
Sequential Tests of Educational Progress grades 3, 6, 9, 11 
Tests of General Educational Development (graduating 
senior, 1966 only; subtests in grammar, literature, 
mathematics, social studies, natural sciences) 

8. Attitude toward various academic subjects (on locally-constructed 

semantic differential instrument) — grades 7, 10, 12 in 
1966 only) 

9. Individual test results and referral records of the school psychologist 

10, Health record and referral records of the school nurse 

11, Attendance (days per semester) for each year 

12, Vocational Interest Checklist — grade 9 only 
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SCIENCE ACHIEVEMENT 
1966-67 



1. Throughout standardized testing is dene in grades 

three, six, nine and eleven, as a part of the State-County Testing 
Program. 

2. In grades three and six the Stanford Achievement Battery is used and 
In grades nine and eleven the Sequential Tests of Educational Progress 
are used. 

3. According to the most recent test data the number of students 
making exceptionally high and exceptionally low scores in science 



is as 


fol lows : 








’ GRADE 


NUMBER OF STUDENTS^ 
SCORING ABOVE THE 
SOth PERCENTILE 
(NATIONAL NORMS) 


NUMBER OF STUDENTS 
SCORING BELOVy THE 
10th PERCENTILE 
(NATIONAL NORMS) 


NUMBER 

TESTED 


OF STUDENTS 
IN EACH GRADE 


3 


623 


573 




5096 


6 


619 


775 




4838 


9 


584 


307 




4891 


11 


371 


292 




4169 
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July 21, I9&7 



Dear Dr. Laytor: 

I'm again being Innundated with requests— by wall-neaning 
people, I'm sure-for meetings with persons who have thj solu- 
iion to problems here in the district. A great number >f 
people have been motivated to react to the Regional ^ 

port. I'd like to react to iU people's suggestions an. requests 
with as much co nsistency as I can, and i'd like to have you do 
some work to aid me in achieving this consistency. , 

materials I've given to you over the pest days, would yo P“ 
together a set of possible objectives for district actioi whjc 
relate to needs and problems you've perceived. Of course, I H 
want information on the degree to which any object ivesVoi suggest 
are presently being attained. Could you, then, give me c list 
of likely mu sources of informat ion for 

formation to' see if we've already attained them) relative to 
each objective you identify? 

I'll check with you Friday on your progress. 



Yours truly, 

Phillip McBride 
Superintendent 



Task 2.0 



August 1967 



< 



Dear Val ; 

Your activities as an evaluator for this past month have 

been extremely helpful to me. Now I'd like 7 °“ J"® 

critical iudament on a ffi job 1 must complete. The Goals 
Task Force has completed a report on objectives which , 

Let the most pressing need of the district J^e ^ject.ves deal 
with a space-related science and planetarium program U 
tached their report). 

I need you to help me t make a program proposal out of these 
ideas, for the corns ide rat ion of the School Board. 
has decided the Task Force's ideas are good, and several Board 
meLer^rre prLsing me to produce a Title 111 Pjoposa based 
on the attached objectives. I'd like to make a l easibl e, as 
well as a relevant , program— lately everyone seems concerne 
about all the aboTtive attempts at change we ve seen in the 
past. What I'd like you to do is select or screen fr^ among 
the attached materials the most feasible objectives for the 
total program. 



Thanks i 



McB. 
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MEMO: The . School System 

TO: All Science Faculty Members 

FROM: Goals Task Force 

RE; . Statement of Need in the Region 



Young people today are maturing in a world where the sum total of 
knowledge will double in one decade# This "explosion of knowledge is 
recognized as a force affecting each of our lives in the last half of the 
20th century# It is also recognized that a major focus of this knowledge 
explosion is space-related science , and that new concepts and technological 
developments in this area not only extend the limits of our environment, but 
also effect changes in its very nature. Thus we are living in the Space Age 
and must seek to understand what implications it has for us as individuals 
and as members of the community# There are economic, sociological, and 
political considerations; a new philosophical orientation is needed; even 
the art forms of our day reflect Space Age influence; to say nothing of the 
vast array of scientific and technological advances that both improve and 
complicate our lives . There are few areas of thought in either the sciences 
or the humanities which have not felt the impact of the Space Age. All of 
which poses a basic problem for the educational systems of our nation. 

They must provide young people with the knowledge and attitudes that will 
enable them to mature adequately and become effective citizens in this 
age of Space and space-related science# 

A curriculum in astronomy and space science was written in the 
summer of 1966 for grades K through 6. It presently is being tested in 
thirteen classrooms in the county. During the summer of 1967, it should 
be evaluated and revised for expansion to other parts of the system# Curri- 
culum writing in space-related science is scheduled for i;he secondary 
level during the summer of 1967 # The limited funds available for both of 
these endeavors, however, does not allow for assistance by specialists 
in space-related science curriculum. The curriculum places little or no 
formal emphasis on the interrelatedness of space science to other science 
and non-science subject areas. Practically speaking, such emphasis has 
been a matter of individual teacher preference and ability * 

An example of this lack of interrelatedness is the poor use of 
existing facilities. For example, during the past two years, ninety-four of 
the 122 elementary schools and seventeen of the thirty-four secondary 
schools have brought an average of four classes to visit the planetarium of 
^he Museum# This indicates not only teacher interest in 
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providing further learning opportunities in the space-related science, but 
also some 888 hours of teaching time devoted to these opportunities. There 
Is, however, only minimal coordination of the planetarium lecture content 
with the classroom curriculum, and the effectiveness of the visit depends 
solely on the ingenuity of the individual teacher to prepare his students 
for their learning experience and to reinforce it through follow-up classroom 
cictivities. In addition, the planetarium is staffed by amateur volunteers 
whose abilities vary tremendously. 

County thus has a need and an opportunity to replace 
current deficiencies in the area of space-related science with a systematically 
structured program emphasizing not only the subject area, but also its 
relationship to other areas of instruction. The program, as proposed, deals 
with the needs for teacher in-service training in content and methods, 
curriculum development, and instructional materials; and stresset. the 
Involvement of students in the experiences of science . 



Objectives to Meet the Local Need 

The objectives of the pilot operation of the Space -related Science 
Project, we believe, ought to be the following: 

A. To develop, during the summer of 1967, space-related science 
curriculum materials for grades K tlirough 9 and to use these materials, 
during the 1967-68 school year, in a pilot program involving approximately 
ten per cent of the school population. The materials will emphasize the 
Interrelatedness of Space Science to other subject areas and the involvement 
of students in the experiences of Science. 

B. To coordinate the use of The Museum Planetarium 

with the space-related science curriculum materials and to Improve its 
capabilities as an educational resource for the schools and the community, 

C. To provide teacher in-service training in space-related science 
content and science teaching methods and materials through (1) a schedule 
of In-service courses for pilot teachers, and (2) Space-related Science Spec- 
ialists whose responsibilities include giving maximum individual assistance 
to the pilot teachers • 

D. To Increase the learning opportunities in space-related science 
through the use of supplementary student activities which extend and enrich 
the curriculum. 

E . To increase the availability of space-related science resource 
materials and equipment to pilot students and teachers . 
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F# To disseminate information about space“related science and 
the Project which encourages positive attitudes on the part of (1) principals# 
teachers, and other school personnel, (2) students and their parents, and 
(3) the community at large • 

G. To maintain an awareness of new knowledge in space-related 
science and recently developed educational practices that may be of use 
In the ongoing Project. 

H. To produce sample television programs in space-related 
science in cooperation with the area educational television broadcasting 
facility. 

I. To gather data and to plan for the step-wise expansion of 
Project services and facilities. 

j, To incorporate a continuing program of evaluation designed to 
assess the degree to which the above objectives are attained and to provide 
Information necessary to the maintenance of Project effectiveness. 
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PROPOSED BUDGET 



[ 



A< Administration 

Salaries (Project and Planetarium Director) 27,000 
Materials 1,500 

Travel 1,000 



B. Instruction 

Salaries 48,500 

Science Specialists (3) @ 11,000 
Evaluation Specialist @11,500 
Secretary @ 4 , 000 

Materials 7,000 

Travel 7,000 

Consultants 4,000 

Stipends for materials development 23,000 

ETU Programming 1,000 

Dissemination 1,000 



C • Planetarium Instruction 
Salaries 
Travel 
Consultants 



13,000 

1,500 

500 



D, Plant Operation (Planetarium) 

E . Pupil Transportation 



F . Equipment 
Office 
PlanetL ium 
Truck 



3,000 

3,000 

3,000 



G* Fixed Costs (Benefits) 
Total 



$29,500 



91,500 



15,000 



1,500 

3.000 

9.000 



8,000 

$157,500 
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August 9, 1967 



Dr. Phillip McBride, 
Superintendent 



Dear Dr. McBride: 

Regarding the possibility of funding your proposed Title III 
program in astronomy and space-related science, I am con- 
cerned about the tentative budget you submitted . I have 
talked to your finance officer, Mr. Dennis O'Harrow, and 
informed him that the estimated cost of the first year of the 
pilot program — $158,000 — is considerably highe*^ than we 
customarily fund such programs. I suggested he seriously 
investigate the possibility of reducing the budget by approxi- 
mately one-half, especially in view of the likely Federal cuts 
in domestic spending during the next fiscal year. I look 
forward to talking with you further as your project develops. 



Cordially 




Will Ketchem, Auditor 
Office of Coordinator 
of Federal Programs 
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Likely Tasks of Personnel 



The Project Director will be responsible for the overall su- 
pervision and administration of Project activities; coordinate 
the curriculum writing conference; Instruct In-service teacher 
training courses; coordinate study and evaluation programs and 
direct revisions and changes In Project operations; coordinate 
planning efforts and Implement new procedures which are devel- 
oped; supervise dissemination of Information; prepare budget and 
exercise Its control; supervise proper maintenance of records 
and preparation of reports and proposals; maintain liaison with 
government agencies (Including state and federal Departments of 
Education); develop areas of cooperation between Project and 
other educational agencies and Institutions; supervise devel- ^ 
opment of educational television programming; supervise acquisi- 
tion of Project library with accurate and up-to-date materials 
on space-related science Information '^nd science education 
pract Ices. 

The Planetarium Specialist will be responsible for the 
operation and maintenance of the planetarium fad 1 Ity; coor- 
dinate planetarium scheduling; assist Space-related Science 
Specialists In preparation of planetarium lessons for pilot 
operation; write and produce public planetarium programs;^ 

Install and maintain auxiliary planetarium equipment; design and 
build new auxiliary equipment, as required; present or supervise 
the presentation of all public programs; disseminate Information 
on planetarium programs to general public and to scheduled 
school groups; maintain accurate, timely knowledge of space-re- 
lated science developments for Incorporation In public program- 
ming; advise on public program and planetarium facility ex- 
pansion; supervise acquisition of and maintain planetarium 
equipment, library of films, slides, auxiliaries, sound 
tapes, and other visual effects; prepare planetarium operating 
budget and supervise Its expenditure; deep accurate records 
of planetarium use, operative checks and maintenance performed; 
assist In acquiring data on audience reaction to public programs. 

The Space- rel ated Science Specialists will be responsible 
for conducting In-service teacher training; provide maximum 
individual assistance to pilot classroom teachers In subject 
content, methods, and materials use; encourage and assist with 
supplementary student activities; arrange for and direct large 
group activities such as “Star Parties” and family programs; 
present three-part planetarium lessons (including pre-vlslt 
orientation and post-visit reenforcement); counsel and advise 
teachers In matters of scheduling and evaluation; disseminate 
Project Information to pilot schools; assist In development of 
educational television programming; maintain accurate, timely 
knowledge of space-related science content and developments in 
science education; advise on expansion of program and educat lon- 
al planning; keep accurate records of Project activities as re- 

D-56 



F 












quired by evaluation procedures; implement revisions in materials 
and procedures resulting from Project evaluation. 



The Evaluator will be responsible for carrying out all the 
evaluation tasks of the Project; work with Project staff to ^ 
explain evaluation techniques and outline record-keeping require- 
ments; design and produce, schedule and administer evaluation 
Instruments; observe and confer with a valid sample of pilot 
school personnel; supervise the processing and analysis of all 
data collected; make weekly reports to Project Director and 
monthly reports to planning groups and The Children s Museum 
ort evaluation results; prepare year-end evaluation report for 
Project staff. Board of Education, state and federal Departments 

of Education. 
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September 5# 196? 



Dr. E. Val Laytor, 

Evaluation Specialist 

Central Administration Building 



Dear Vsi : 

I don't know if you are in a position to do anything about it, but 
l*m going to complain to you anyway. I hope I'm not misusing our 
friendship. 

I called McBride as soon as I heard about your district's decision 
to put all their eggs into that misbegotten Title III planetarium 
project. I told him that I couldn't conceive of a focus much less 
related to what we see as real needs in this area. But I guess he 
isn't going to listen. 

Val, !'m really disturbed. We spent a lot of time and research on 
our report on "regional goals for the 1980's." We identified many 
problems that are crucial in Appalachia--and Charleston is no excep- 
tion. We have hundreds of kids who can't read their names, and you 
guys decide to teach them about the stars! I just don't understand 
the logic, Can't you get to McBride and talk some sense to him? 

If I can help you convince him, let me know. 



Sincerely, 




Wayne *Newton 
Research Director 
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August 12 f 1967 



Dear Val: 

Thanks for doing such a thorough job of scrutinizing the 
proposed objectives for the Title III space -related science 
program. The over-all objectives the Board decided upon are 
attached • 

The one objective I*m really concerned about is the one 
regarding in-service education. I’d like you to help me look 
at how we might best attain this objective. Would you set 
some criteria for selecting among alternative programs? Then, 
use your creativity and the criteria to generate alternative ways 
to attain the in-service education objective. Can you give me 
the criteria and a list of alternative programs (just one or two 
sentences describing the program, and a notation of the criteria 
which you expect the program to meet) by next ¥/ednesday? 



Thanks, 




Phillip McBride 
Superintendent 



PM;jt 
Task 4,0 



D-60 



Obiectives 



Th© objectiv6S of th© first year pilot operation of the Space-related 
Science Project are: 

A, To develop, during the summer of 1967, space-related science 
vcurriculum materials for grades K through 9 and to use these materials, during 
the 1967-68 school year, in a pilot program involving approximately ten per 
cent of the school population. The materials will emphasize the interrelated- 
ness of Space Science to other subject areas* and the Involvement of students 
In the experiences of Science, 

B, To coordinate the use of The Children* s Museum Planetarium with 
the space-related science curriculum materials and to improve its capabilities 
as an educational resource for the schools and the community, 

C , To provide teacher in-service training in space-related science 
content and science teaching methods and materials through (1) a schedule 
of in-service courses for pilot teachers, and (2) Space-related Science 
Specialists whose responsibilities Include giving assistance to the pilot 
teachers , 

D , To Increase pilot teacher knowledge of the availability of space- 
related science resource materials and equipment, 

E, To disseminate information about space-related science and the 
Project which encourages positive attitudes on the part of (1) principals, 
teachers, and other school personnel, and (2) students and their parents, 

F, To maintain awareness of new knowledge in space-related science 
and recently developed educational practices that may be of use in the 
ongoing Project, 

G , To gather data and to plan for the step-wise expansion of Project 
services and facilities, 

H, To incorporate a continuing program of evaluation designed to assess 
the degree to which the above objectives are attained and to provide informa- 
tion necessary to the maintenance of Project effectiveness. 



* See attached "Space-Related Science" diagram. 
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SPACE-RELATED SCIENCE 

EMBRACES MANY DISCIPLINES 



General Science 

Biology 

Chemistry 

Physiology 

Earth Science 

Physics 



History 
Literature 
Social Sciences 
Civics 
Sociology 



The Sciences 



The Humanities 




Mathematics 

Arithmetic 

General Mathematics 

Algebra I, II, College 

Geometry, Plane, Solid 

Trigonometry 

Analytic Geometry 

Calculus 

Statistics 



Communication Arts 

Reading 

Writing 

Speech 

Art 

Graphics 
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College of Education 



sar Colleague in Evaluation: 

have recently been looking at the problems of educational planning and the assess- 
ent of ^ternatlve strategies for attaining goals which educators deem desirable, 
t seems to be true throughout education that there is I Ittle systematic caparison 
f wa^ to attain the objectives which local schools, whole districts, or larger 

ystems state are of concern to them. 

believe that some ideas I have been working with might provide a framework within 
hich the assLI^nt of Inputs might be viewed. The following are criteria ^ich 
have foun“heir-ne evaluate alternatives; the criteria are vague, and may be used 
oSetLr or separately, with varying degrees of importance attached to them, but 
hey help me appear more rational when reasonableness is neede . 



RITERIA FOR EVALUATING ALTERNATIVE STRATEGIES FOR ATTAINING EDUCATIONAL OBJECTIVES: 



[elevance : determining the relevance of the proposed solution strategy to 

■eoal I tv "*^’ determining the’ legal status of the proposed solution relative to 
the context In which it is to be implemented; 
ionaruence: determining the congruence of the solution with the value sy 

tiaTs) of the agency charged with implementation; ^ 

LeQltlmacy : determining if the solution is within the purview of the 

agency charged with the implementation, w-»ino 

Comoatability : determining the corapatabi 1 ity of the strategy with the value 

system(s) of the implementing agency and its staff, 

Balance: determining the impact of the solution on other co^onents of the 

system of concern, and on the weights and interrelationships of the 

PractUabHUv:^ determining the practicability of the solution in terms of 
■ achieving its stated purposes (end products or outcomes); 
rnst-affactiveness : determining the relative desirability of the 

In comparison with other solutions, in terms of the ratio of necessary 
Inputs (costs) and outputs (effectiveness). 



My hope is you will use and revise them at will. 
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Michael S. Caldwell 
Associate Director 
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Dr. E . Val Laytor 
Evaluation Specialist 

Central Administration Building 



Dear Dr. Layton: 

I've had the members of the Goals Task Force work over the 
objective that you and Supt. McBride were concerned about. 
To make what we intended more apparent, we have restated 
the objective and articulated a set of sub-objectives. Our 
thinking was that we would assume the objective was attained 
when we could see evidence that the sub-objectives had been 
attained. I hope these ideas give you some lever on the job 
of creating different ways of going about attaining our in- 
service education objectives. 



Yours truly 




John Blaha, Chairman 
Goals Task Force 



Attachment 
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Major and Minor Objectives of the In-Service Education Program: to 
provide teacher in-service training in space-related science content^ 
methods, and materials (and to attain this objective through the 
following); 

A) In-service courses for pilot program teachers (v/ith 

sub-objectives being); 

1) to present five training courses which utilize the five 
levels of curriculum materials being developed; 

2) to familiarize teachers with the curriculum materials 
in space-related science; 

3) to demonstrate the use of the inquiry approach with 
these curriculum materials; 

4) to familiarize teachers with available services, materials, 
and equipment which might be used in the space -related 
science program; 

5) to generate teacher interest and enthusiasm in the 
adoption of these curriculum materials . 

B) Individual assistance provided by science specialists in the 

program (with sub- objectives being): 

1) to answer teachers' subject matter inquiries; 

2) to keep teachers informed of current developments 
in the subject-matter area of space-related science; 

3) to provide resource equipment and materials on request; 

4) to demonstrate inquiry teaching methods on request; 

5) to engage in team teaching in space-related science 
on request 

6) to assist v/ith small-group investigations on request. 



Task 4,2 



D-65 



A period of several months' time has elapsed since you 
worked with designing alternative in-service education programs 
(in the previous simulation). During the past months the draft 
proposal was prepared in its final form and submitted , and a 
Title III grant received for establishing and operating a space- 
related science program. The objectives of the program are 
those with which you are familiar (and were listed in the 
previous simulation). The program is now well under way, 
having been started at the beginning of the school year . Your 
task is to monitor the operation of the program's activities and 
detect deviations from the manner in which the program was 
expected to operate. You have been monitoring potential 
barriers or sources of trouble which you predicted might impede 
program activities, as well as remaining alert to unanticipated 
barriers to program success. Periodically you have been sending 
reports to decision-makers responsible for the program. Then — 
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PROCESS EVALUATION REPORT NO. 14 



Project ; In-Service Education 
Program; Space-Related Science, Title III 
Date ; November 20, 1967 

Data Collection and Report Preparation; E. Val Laytor 
SUBJECT; Attendance at In-service Education Meetings 

Criterion; 

Fluctuation^ in teacher attendance at in-service education meetings was 
one of the factors identified during project planning as a likely barrier to 
project success. It was believed ^hat a serious drop in session attendance 
would make it unlikely that project objectives could be fully attained. 

Findings and Interpretation 

During the past four meetings, there has been a marked and consistent 
downward shift in attendances The following figures illustrate this; 



Initially, 75 persons were expected at the sessions; it was anticipated that 
attendance would remain at that level. Thus, it appears unlikely that the 
desired level of teacher training will be achieved unless steps are taken to 
insure a high level of regular repeat attendance • 

Data Collection Organization and Analysis ; 

Session attendance was taken by counting entrants to the in-service 
training sessions. The evaluator recorded the actual attendance figures 
on a roll card, and checked project files to determine expected enrollments. 
Analysis involved comparison of expected and observed attendance by meeting . 



Meeting 



Attendance 



1 (Oct. 27) 

2 (Nov. 3) 

3 (Nov. 10) 

4 (Nov. 17) 



74 

63 

55 

46 
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MEMO: 



FROM: Sally Richardson 



Dear Teacher: 

I am happy to hear that you have agreed to be one of the 
pilot teachers in the Space-Related Science Project* As you know, we 
will hold a special in-service course to provide training in space- 
related science content, methods, and materials * 

The 'bourse will be held in the multi-purpose room at Fairview 
Elementary Sciiocl on West Seneca Avenue o rne instructor will be 
Mr. V/illard Leer, the new Director of the Space-Related Science Project, 

A schedule of the sessions and topics appears below. Each 
session meets for one hour. 



October 27 



Introduction to Space -related Science 
Curriculum Materials 



November 3 



Developing Student Investigations in 
Space-related Science 



November 10 



Techniques for Using Astronomical 
Equipment 



November 17 



Space -related Science and the Inquiry 
Approach 



November 24 



Space-related Science Curricula: Units 1-3 



December 1 



Space-related Science Curricula: Units 4-6 
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You've been so immersed in the many activities of 
an evaluator that you have- completely overlooked beginning 
a job that you should have completed some time ago. You 
have not yet made a product evaluation design for either 
the program as a whole or any of its parts. This has been 
brought painfully to your attention by a reminder to you 
(in the hallway) from Supt. McBride, who observed he 
would soon be ready to "take a hard look at the in-service 
training program" to see if further programs ought to be 
conducted in the same way • 

You've decided to begin the task by developing a 
product evaluation design for five sub-objectives of the 
in-service program* Having constructed the attached work- 
sheet, your job now is to flj.l in the 'worksheet* 
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Product Evaluation Worksheet #1 
Program: In-Service , part A 



Sub-obi active s 



Criteria 

(Acceptable Evidence) 



Information 

Needed/Source 



Instrumei 



er|c 



1 . To present five training 
courses which utilize the 
five levels of curriculum 
materials being developed 



2 . 



To familiarize teachers 
with the curriculum 
materials in space- 
related science 



3 • To demonstrate the use of 
the inquiry approach 
with these curriculum 
materials 



To familiarize teachers 
with available services , 
materials, and equipment 
which might be used in the 
space-related science 
program 



To generate teacher interest 
and enthusiasm in the 
adoption of these curriculum 
materials 






U 



C' 



O M 



ation 
Source 



Instruments 



Analyzing 



Report 



v\ 




i 



( 

f 

f 




January 16, 1968 



Dr. E. Val Laytor 

Central Administration Building 



Dear Dr. Laytor: 

TO respond to your SL met, I dis- 

ruSerrm=%r?^^^^ conea- - 

in the in-service training program J^VdluberrUon we decided the following 
:e^^\"e"sTviTer/- be comfortable accepting f«.m others and defendxng 

to the public: 



Objectives 

To present five training courses 
which utilize the five levels 
of curriculum materials being 
developed 



Acceptable Evidence 



Content validity of the training course in 
relation to the level of curriculum materials 

discussed 



2 . To familiarize teachers with 

the curriculum materials in 
space-related science 

3. To demonstrate the use of the 

inquiry approach with these 
curriculum materials 

4. To familiarize teachers with 

available services, materials, 
and equipment which might 
be used in the space-related 
science program 



95% repeat attendance at sessions 

Coverage of the entire curriculum unit in 
the six hours of sessions 

Minimum of one use of method for each 
unit at each level of curriculum materials 

Ability to deal with hypothetical situations 
by obtaining relevant resources 

Expressed knowledge about services, 
materials , and equipment 
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January 16 , 1968 



Objectives Acceptable Evidence 

5 • To generate teacher interest and Positive shift in teacher attitude 
enthusiasm in the adoption of 

these materials Teacher level of use or non-use of 

curriculum materials 

If we may be of further help to you, please feel free to contact me. We look forward 
to reviewing your forthcoming reports concerning the impact of our district's 
Title III educational programs on the needs of this area. 

Sincerely yours. 




Chairman, 

County Board of Education 
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January 18, 1968 



Dr. E . Val Laytor 

Central Administration Building 



Dear Dr. Layton: 

I understand you’re planning an evaluation of the space-related science 
program to assess the extent to which we got the payoffs we hoped for. 

I know evaluation is a difficult job, and I want to help you in any way 
I can. I'd be glad to discuss with you what the program has done; I've 
been with the program since it was funded, and am familiar with all 
aspects of it. It seems that it would be easier on teachers and staff 
members in the program, as well as more productive for you, to use 
materials I have here in the office and reactions and perceptions I've 
gotten from working with people ifi the program. If you'd find it helpful, 
I'll be glad to stop by and work with you. 



Yours truly 




Willard Leer, Director 



Task No . 6.3 



D-73 



You have finally completed the product evaluation 
design. The last major task facing you is that of 
drawing up a master schedule for the collection of the 
information (administration of instruments, etc.). 

Start by drawing up a master schedule for the data 
collection in the portion of the design which you have 
just completed (the product evaluation of the first five 
sub-objectives of the in-service training program). 
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January 28, 1967 



< 



Dr. E. Val Laytor 

Central Administration Building 



Dear Val: 

I've just received your note requesting a series of interviews with project personnel 
in the Title III space -related science program. Feel free to begin your work any 
time; there is no problem in interviewing these people. Your request reminded me 
of the problem we had with Judd Mental's scheduling of evaluation activities, 
when he was here as your predecessor — and because of those problems I ought 
to fill you in on some general guidelines for testing, etc., here in the district. 

The following guidelines are the outgrowth of discussions between myself and 
staff ©f the district: 



a) Students may be tested only during the first two and last two weeks 
of the school year 

b) Teachers may be used as information sources only under their own 
volition 

c) Project personnel are required to provide evaluation information needed 
by the evaluator 

d) All other personnel may serve as information sources only after the 
matter is approved by the central administration. 

I think it would be best to use these guidelines in setting up any further work you dc 



Yours truly, 

Phillip McBride, Supt. 
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SOME METHODS OF DATA COLLECTION 



1. Objective tests 

2, Essay tests 

3« Projective methods 

4. Rating scales , Check lists. Inventories 

5 . Semantic differential 

6 . Questionnaire 
7 « Interviews 

8. Observation of behavior 
9 • Sociometry 

10, Unobtrusive measures and content analysis 




unavailable through validity 

other techniques 



RATING SCALES , CHECK LISTS , INVENTORIES 
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ERIC 



SEMANTIC DIFFERENTIAL 





leadership 



EXAMPLE OF INTERVIEW 
OR QUESTIONNAIRE 



In g©n©ral/ how would you rate th© program of the health centers as a 
means of providing the necessary medical and dental care for low income 
students ? 

Very good, no problems 

Good, some problems 

Inadequate, many problems 

Very poor, does more harm than food 

Other, please specify 



2, Have you been able to visit the health center that serves your school? 

Yes No 

If yes, are the physical facilities adequate? 

Yes No 

If no, what is needed? 



3, V/hat problems, if any, do your children have in getting to the health 
center? 

Lack of transportation 

Geographical location 

Inability of parent to accompany child 

Parental Indifference 

Poor home-school communications 
Other, please specify 



4 • Have you been able to form a clear perception of your role in relation 
to the Health Centers Project? 

Yes No 

If yes, please describe this role, (Functions and responsibilities) 



o 
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EXAMPLE OF A 
SOCIOMETRIC ITEM 



Write the names of the three children in the class you would most 
like to have sit near you , 



1 . 



2 . 



3. 
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EXAMPLE OF 



CHECK LIST ITEMS 

Please check those pieces of equipment which you 
frequently use in your science classes: 

Record player 

Tape recorder 

Filmstrip projector 

Overhead projector 

Film projector 

EXAMPLE OF 
INVENTORY ITEMS 

1, Select from the three activities listed# the one that you 
would most like to perform: 

a . Design new styles for automobiles 

b. Conduct an advertising campaign for a particular 
make of car 

c . Sell automobiles 

2 . I would rather (choose only one): 

a . Burn my draft card 
b • Deprive a baby of food 
c • Steal money from a church 

D-84 
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SELECTED DATA COLLECTION METHODS 



INFORMATION NEEDS 

A. PSYCHOLOGICAL 
VARIABLES 

1 . Intelligence 
Z . Aptitude 

3 • Achievement 

4 • Personality 

5 • Readiness 

6 • Sensory Motor 

7, Vocational preference 

8. Interest 

9 • Attitudes/Values 

10. Motivation 

11. Creativity 

B. SYSTEMS 
VARIABLES 

1 • Personnel 
2 • Finance 

3. Facilities & materials 

4 • Communications 

5 • Structure 

6 • Influences 

7. Content/systems output 


Objective tests 


Essay tests 


Projective methods 


Rating scales 


Check lists 


Inventories 


Semantic differential 


Questionnaires 


Interviews 


Observation of behavior 


Sociometry 


Unobtrusive measures 


Others 
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8. Goals & objectives 
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ERIC 



Assessing Potential 
Barriers to Success 



APPENDIX TABLE 1 



RESPONSES BY TEACHERS AND SUPERVISORS 
TO STATEMENTS ABOUT THE INSTITUTE 



Item 

Number 


SD 


Teachers 
D N A 


SA 


j 

SD 


Supervisors 
D N A SA 


Total" Group 
SD D N A SA 


i: 








6 


4 






1 


29 


33 






1 


35 


27 


2. 








2 


8 






1 


8 


44 






1 


10 


52 


3, 


6 


4 




1 




28 


20 


3 


2 




34 


24 


3 


3 




4. 




1 




8 


2 




1 


2 


27 


22 




2 


2 


35 


24 


5. 




1 


1 


7 


1 


1 


3 


3 


43 


3 


1 


4 


4 


50 


4 


6. 








10 


1 


1 


1 


3 


38 


10 


1 


1 


3 


48 


11 


7. 






1 


8 








8 


32 


12 






9 


40 


12 


8. 






1 


5 


5 




2 


5 


25 


20 




2 


6 


30 


25 


9. 






1 


9 


1 


1 


1 


7 


39 


5 


1 


1 


8 


48 


6 


10. 








10 




1 


2 


8 


35 


6 


1 


2 


8 


45 


6 


11. 






3 


7 


1 




2 


10 


35 


5 




2 


13 


42 


6 


12. 






1 


9 


1 




2 


10 


35 


5 




2 


11 


44 


6 


13. 








8 


3 






3 


39 


10 






3 


47 


13 


14. 




1 


2 


6 


2 






8 


42 


2 




1 


10 


48 


4 


15. 








8 


3 




1 




34 


17 




1 




42 


20 


16. 






2 


6 


3 




1 


8 


36 


7 




1 


10 


42 


10 


17. 






2 


9 






4 


11 


34 


3 




4 


13 


43 


3 


18. 






1 


10 




1 


6 


9 


19 


16 


1 


6 


10 


29 


16 


19. 






2 


8 


1 


1 


2 


9 


27 


12 


1 


2 


11 


35 


13 


20. 


2 


4 


3 


1 




7 


22 


16 


6 




9 


26 


19 


7 




21. 






2 


7 


2 




1 


5 


29 


17 




1 


7 


36 


19 


22. 


2 


4 


3 


2 




4 


26 


18 


11 


2 


6 


30 


27 


18 


2 


23. 


2 


7 


2 






9 


28 


13 


2 




11 


35 


15 


2 




24. 


2 




2 


5 


2 




3 


13 


33 


4 


2 


3 


15 


38 


6 


25. 


2 


6 


3 






6 


28 


13 


4 


1 


8 


34 


16 


4 


1 


26. 




5 




4 


1 


11 


29 


6 


5 




11 


34 


6 


9 


1 


27. 


2 


9 








15 


34 


3 






17 ' 


43 


3 






28. 


3 


7 


1 






16 


34 


1 


1 




19 ^ 


41 


2 


1 




29. 








6 


5 


1 




1 


14 


36 


1 


1 


1 


20 


41 


30. 


2 


3 




6 




3 


11 


4 


24 


10 


5 


14 


4 


30 


10 


31. 


1 


6 


1 


2 


1 


8 


31 


9 


4 




9 


37 


10 


6 


1 


32. 


4 


6 


1 






9 


28 


7 


5 


2 


13 : 


34 


8 


5 


2 


33. 






5 


4 


2 






14 


32 


6 






19 


36 


8 


34. 








6 


5 






1 : 


23 


28 






1 : 


29 


33 


35. 


4 


6 


1 






10 


31 


4 


7 




14 ; 


37 


4 


7 




36. 


3 


5 


1 


2 




7 


27 


5 


14 




10 : 


31 


6 


16 




37. 




1 


5 


5 




1 


11 


14 ; 


24 


2 


1 


12 


19 ; 


29 


2 


38. 




1 


1 


7 


1 




2 


18 : 


27 


4 




3 


19 : 


34 


5 


39. 








4 


7 






5 


18 


29 






5 ; 


22 


36 


40. 


2 


6 




3 




6 


28 


13 


3 


1 


8 ; 


34 


13 


6 


1 


41. 


4 


5 




2 




25 


25 




1 


1 


29 ; 


30 




3 


1 


42. 








5 


6 






1 : 


25 


25 


1 




1 ; 


30 


31 


43. 


1 


9 


1 






5 


33 


10 


4 




6 42 ; 


11 


4 




44. 




8 


2 




1 


4 


25 


17 


5 


1 


4 33 ; 


19 


5 


2 


45. 


1 


5 


4 


1 




6 


29 


6 


7 


4 


7 34 : 


10 


8 


4. 


46. 






2 


5 


4 

4 


A A • 




4 27 


21 






6 ; 


32 


25 



SD=strongly disagree; D=disagree; etc. 
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APPENDIX TABLE 2 



EVALUATIONS BY TEACHERS AND SUPERVISORS 
OF SPECIFIC ACTIVITIES BY DAYS 



Days 


"’oo 

Much 


Teachers 

About 

Right 


Too 

Little 


Supervisors 
Too About Too 

Much Right Little 


Total Group 
Too About Too 

Much Right Little 


First 




















Day 




















1. 




8 




1 


43 


3 


1 


51 


3 


2. 


2 


5 




4 


38 


9 


6 


43 


9 


3. 




6 


3 


2 


37 


13 


2 


43 


16 


4. 




4 


4 


3 


21 


28 


3 


25 


32 


Second 




















Day 




















1. 


3 


8 




14 


24 


14 


17 


32 


14 


2. 


1 


3 


7 


10 


19 


23 


11 


22 


30 


3. 


1 


4 


6 


8 


20 


24 


9 


24 


30 


4. 




2 


8 


8 


15 


28 


8 


18 


36 


5. 




2 


8 


8 


14 


30 


8 


16 


38 


Third 




















Day 




















1. 


2 


9 




3 


44 


5 


5 


53 


5 


2. 


3 


5 


3 


9 


21 


20 


12 


26 


23 


Fourth 




















Day 




















1. 


1 


7 


2 


3 


38 


12 


4 


45 


14 


2. 


2 


6 


2 


4 


30 


19 


6 


36 


21 


3. 


2 


6 


2 


5 


32 


16 


7 


38 


18 


4. 




10 




1 


48 


2 


1 


58 


2 


Total : 


1 


8 


2 


3 


32 


18 


4 


40 


20 



